PROBLEM: Request line too large - user problem / workflow problem / OpenAlex setting - how to solve it?
36 views
Skip to first unread message
Rainer M Krug
Nov 12, 2025, 8:06:27 AM11/12/25
to OpenAlex Support, OpenAlex Community
Hi
I am writing to support as well as the group, because I see this an important point for the usage of OpenAlex via the API.
Usage scenario: We are conducting a literature search with a complex search terms. Below is one API call.
This does not return results, as the request is to large (Request Line is too large (4195 > 4094)).
Now. I have several questions concerning that:
1. I (data expert role) am working together with the topic experts to obtain the number of hits of this search. We had several back-and-forth concerning the length problem, but it is occurring again and again (as the search term conists of several sub-searches combined by AND and they are edited independently. They would like to know if they can have a idea what is influencing the length of(double inverted commas `”`, etc) and if there is a way to get this number quickly from the search term without having to do the API call? Especially with complex search terms, this can take a few seconds. It would be great to get clarification on these issues.
2. I assume that this issue will come up again and again. It would be great, if it would be possible that this limit can be increased - I hoped Wals=den could solve it, but I assume it comes from an underlying component of the API.
3. Considering that the final term consists out of at least 3 independent sub search terms which are combined by AND, What are alternative approaches? I could get the ids only via `select = ids`, do local AND which would work - but each of the sub search terms may return millions of records and that would take a considerable time (due to the paging). Are there any suggestions? And no - I can’t use a local snapshot as I am doing full text search and want to stay in-line with the OpenAlex website.
Oh - I am working from R, using openalexPro as well as openalexR.
Thanks a lot for reading till the end,
Rainer
---
The information in this e-mail, and those ensuing, is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, please destroy this message and notify the sender immediately.
 | Rainer Krug Senior Data Specialist SIB | Swiss Institute of Bioinformatics CH - Switzerland Mobile +41 78 630 66 57 Raine...@sib.swiss www.sib.swiss |
Samuel Mok
Nov 13, 2025, 8:40:28 AM11/13/25
to Rainer M Krug, OpenAlex Support, OpenAlex Community
Hi Rainer,
First: heads up -- you included your API key in the url you just pasted. I recommend to disable this key and create a new one to prevent abuse.
Second: I think this is not really a good fit for the API; e.g. as mentioned in the docs OA does some parsing of text into n-grams, which can lead to some misses. You could move to using the lens, as it includes the full OpenAlex dataset plus more, and it can handle the query without changing it (although the link here might break due to the length).
But as an information specialist I would highly suggest finetuning the query before figuring out technical alternatives. I don't think this is an effective query! I'm not sure what the goal is here, but I expect MANY irrelevant results due to the extremely broad list of accepted terms -- and there are quite some removal opportunities besides that:
First: heads up -- you included your API key in the url you just pasted. I recommend to disable this key and create a new one to prevent abuse.
Second: I think this is not really a good fit for the API; e.g. as mentioned in the docs OA does some parsing of text into n-grams, which can lead to some misses. You could move to using the lens, as it includes the full OpenAlex dataset plus more, and it can handle the query without changing it (although the link here might break due to the length).
But as an information specialist I would highly suggest finetuning the query before figuring out technical alternatives. I don't think this is an effective query! I'm not sure what the goal is here, but I expect MANY irrelevant results due to the extremely broad list of accepted terms -- and there are quite some removal opportunities besides that:
direct duplicates, like trade-offs and tradeoffs; world views and worldviews; agro-forest and agroforest; marshes and marshland, etc
different spellings, like "spatial conservation prioritisation" and "spatial conservation prioritization"
redundant phrasing: the query often specifies (term) AND planning or (term) AND management. It is more efficient to search for the complete phrase, such as "adaptive planning" in this case
unnecessary quotation marks: single words that are not part of a specific phrase (e.g., "coastal", "agricultural") do not require quotation marks. if you're worried about stemming adding additional hits, turn it off!
different spellings, like "spatial conservation prioritisation" and "spatial conservation prioritization"
redundant phrasing: the query often specifies (term) AND planning or (term) AND management. It is more efficient to search for the complete phrase, such as "adaptive planning" in this case
unnecessary quotation marks: single words that are not part of a specific phrase (e.g., "coastal", "agricultural") do not require quotation marks. if you're worried about stemming adding additional hits, turn it off!
As mentioned, many terms are very similar or fall under a broader category: bog, mires, peatland, and marshes for example, or coast, sea, ocean, and marine. Also there are quite a few very broad terms, like 'earth', 'planet', etc. The chance a paper you're interested in will mention 'earth' but not any of the other nature-related terms is extremely slim.
Doing a quick check with the original query without any of the filters gives me ~7500 results in the frontend, and deleting a whole bunch of terms related to specific biomes gives me ~6800 results -- not a huge difference for such a drastic change in the query.
When checking the query, I formatted it into a shape that's better readable, I included it at the end of the message for reference. I would suggest to first try a way more simple query, like this one here:
(
(
"adaptive management" OR "conservation planning" OR connectivity OR "ecological corridor" OR "ecosystem based management" OR "ecosystem services" OR "environmental assessment" OR "land use planning" OR "landscape planning" OR "marine spatial planning" OR "nature-based solutions" OR OECM OR IPBES OR rewilding OR restoration OR "scenario analysis" OR "scenario planning" OR "spatial planning" OR zoning
) AND (
agricultural OR biodiversity OR coastal OR ecosystem OR environment OR forest OR freshwater OR habitat OR landscape OR marine OR mountain OR "natural resource" OR "protected area" OR sustainable OR terrestrial OR wetland OR wildlife
)
) AND (
(
"indigenous knowledge" OR "local knowledge" OR biocultural OR customary
) AND (
governance OR planning OR policy OR rights OR tenure
) AND (
equitable OR equity OR fair OR inclusive OR just
)
)
And then adding terms step-by-step and see how much extra items you're catching by adding the terms, and taking a look to see if the added items are valuable to the dataset. Iteratively building up the query will let you streamline which terms make sense to keep, and which ones are redundant or even damaging to your results.
The original query, formatted by me:
The original query, formatted by me:
(
(
(
adaptive AND (
management OR
planning
)
) OR (
animal AND (
migration OR
movement
)
) OR (
(
anticipatory OR
"community based" OR
"forest management" OR
"land-sea" OR
participatory OR
"place based" OR
regional OR
sectorial OR
territorial OR
urban OR
"agricultural management"
) AND
planning
) OR (
biodiversity AND
(
assessment OR
indicators OR
scenario
)
) OR (
(
blue OR
green
) AND
infrastructure
) OR (
connectivity
) OR (
conservation AND
(
practice OR
planning OR
program OR
strategy
)
) OR (
"critical areas for biodiversity"
) OR (
(
"cumulative impact" OR
"environmental impact" OR
"strategic environmental"
) AND assessment
) OR (
ecological AND (
corridor OR
networks
)
) OR (
(
"ecosystem based" OR
"sea use"
) AND
management
) OR (
"ecosystem service" AND (
mapping OR
model OR
planning
)
) OR (
(
functional OR
working
) AND
landscapes
) OR (
"futures thinking"
) OR (
(
"generalized dissimilarity" OR
macroecological
) AND
model
) OR (
(
habitat OR
wildlife
) AND
corridor
) OR (
(inclusive OR
marine
) AND
governance
) OR (
integrated AND (
"assessment framework" OR
"assessment model" OR
"coastal zone management"
)
) OR (
(
integrative OR
"trends and"
) AND
scenarios
) OR (
IPBES
) OR (
(
"key biodiversity" OR
protected OR
"remote ocean"
) AND
areas
) OR (
land AND (
allocation OR
governance OR
system
)
) OR (
(
"land use" OR
"sea use" OR
marine
) AND (
decision OR
planning OR
governance OR
management OR
model OR
scenario OR
trade-offs OR
tradeoffs
)
) OR (
landscape
AND (
complexity OR
planning OR
governance OR
management OR
"ecological planning"
)
) OR (
"multi-criteria decision analysis"
) OR (
nature AND (
"based solutions" OR
futures
)
) OR (
OECM
) OR (
planning AND (
"for ecosystem services" OR
process OR
scenario OR
tools
)
) OR (
"reserve design"
) OR (
restoration
) OR (
rewilding
) OR (
scenario AND (
analysis OR
"based model" OR
"based planning" OR
building OR
planning OR
thinking OR
"and models" OR
"of land use"
)
) OR (
spatial AND (
composition OR
configuration OR
"conservation prioritisation" OR
"conservation prioritization" OR
decision OR development OR
"forest planning" OR
governance OR
planning OR
prioritisation OR
prioritization OR
transformation
)
) OR (
"stepping stones"
) OR (
"systems integration"
) OR (
zoning
)
) AND (
(
(
"agricultural" OR
"arable"
) AND
"land"
) OR
"agro-forest" OR
agroforest OR
biodiversity OR
bioeconomy OR
biosphere OR
bog OR
chaparral OR
coast OR
"coastal" OR
conservation OR
"coupled system" OR
desert OR
dryland OR
Earth OR
ecological OR
ecosystem OR
environment OR
"environmental resource" OR
estuary OR
fauna OR
fjord OR
flora OR
forest OR
freshwater OR
grassland OR
habitat OR
heathland OR
lake OR
landscape OR
marine OR
maritime OR
marshes OR
marshland OR
meadow OR
mires OR
mountain OR
"natural resource" OR
nature OR
ocean OR
peatland OR
planet OR
plantation OR
(
(
"protected" OR
conserved
) AND
"areas"
) OR
territories OR
"resource system" OR
restoration OR
river OR
savanna OR
sea OR
seascape OR
shrubland OR
soil OR
species OR
sustainable OR
terrestrial OR
tundra OR
wetland OR
wildlife OR
woodland
)
) AND (
(
(
"indigenous" AND (
"knowledge" OR
"perspectives" OR
"governance" OR
"rights" OR
"values" OR
"practices" OR
"land" OR
"territory"
)
) OR
"local knowledge" OR
"biocultural" OR
"customary" OR
(
"diverse" AND (
"knowledge systems" OR
"world views" OR
"worldviews"
)
)
) AND (
"planning" OR
"governance" OR
"policy" OR
"tenure" OR
"rights"
) AND (
"just" OR
"inclusive" OR
"fair" OR
"effective" OR
"equity" OR
"equitable"
)
)
--
You received this message because you are subscribed to the Google Groups "OpenAlex Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openalex-commun...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/openalex-community/F5A8C5F3-5B0D-449A-9FA7-6D44DBCC103F%40krugs.de.
Rainer M Krug
Nov 13, 2025, 8:59:44 AM11/13/25
to Samuel Mok, OpenAlex Support, OpenAlex Community
Hi Samuel
On 13 Nov 2025, at 14:40, Samuel Mok <sam...@gmail.com> wrote:Hi Rainer,
First: heads up -- you included your API key in the url you just pasted. I recommend to disable this key and create a new one to prevent abuse.
Thanks - I will see to this (and thanks for sharing it again at the end). :-)
Second: I think this is not really a good fit for the API; e.g. as mentioned in the docs OA does some parsing of text into n-grams, which can lead to some misses. You could move to using the lens, as it includes the full OpenAlex dataset plus more, and it can handle the query without changing it (although the link here might break due to the length).
Thanks - interesting - I will con look at it. But aren’t the n-grams only for full text? I am only searching for title and abstract, soo it should not be a concern?
But as an information specialist I would highly suggest finetuning the query before figuring out technical alternatives. I don't think this is an effective query! I'm not sure what the goal is here, but I expect MANY irrelevant results due to the extremely broad list of accepted terms -- and there are quite some removal opportunities besides that:
Yes - without doubt it is a highly inefficient query, which will return many irrelevant results. The aim is to be, in this first step, inclusive. And the search term is developed b y many spatially separated expert, most of them have no deep understanding of search terms. So this is something I can live with for the moment.
Also, the search term should be reviewable for the experts at anytime (and fine unable). So a re-structuring would include an additional manual step in-between.
direct duplicates, like trade-offs and tradeoffs; world views and worldviews; agro-forest and agroforest; marshes and marshland, etc
different spellings, like "spatial conservation prioritisation" and "spatial conservation prioritization"
Please try it out - all these you mentioned return different number of results in the title and abstract search (which we are using).
redundant phrasing: the query often specifies (term) AND planning or (term) AND management. It is more efficient to search for the complete phrase, such as "adaptive planning" in this case
unnecessary quotation marks: single words that are not part of a specific phrase (e.g., "coastal", "agricultural") do not require quotation marks. if you're worried about stemming adding additional hits, turn it off!
We removed the inverted commas around individual words, but the results are the same, as stemming is also done for terms in the inverted commas - but it shortened the search string so that we can now run it. But I remember that there were cases, where the search interpreted A AND B differently from “A” AND “B”, i.e. the first one as `A B` (stop word removed), the second one correclly as A as well as B. I remember reading that as well, but do not remember where.
As mentioned, many terms are very similar or fall under a broader category: bog, mires, peatland, and marshes for example, or coast, sea, ocean, and marine. Also there are quite a few very broad terms, like 'earth', 'planet', etc. The chance a paper you're interested in will mention 'earth' but not any of the other nature-related terms is extremely slim.Doing a quick check with the original query without any of the filters gives me ~7500 results in the frontend, and deleting a whole bunch of terms related to specific biomes gives me ~6800 results -- not a huge difference for such a drastic change in the query.
As I said - we are more concerned about not missing any, then returning to many.
Thanks again for your insight - very much appreciated,
Cheers,
Rainer
---The information in this e-mail, and those ensuing, is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, please destroy this message and notify the sender immediately.
Rainer Krug
Senior Data Specialist
SIB | Swiss Institute of Bioinformatics
CH - Switzerland
Mobile +41 78 630 66 57
Raine...@sib.swiss
www.sib.swiss
--
You received this message because you are subscribed to the Google Groups "OpenAlex Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openalex-commun...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/openalex-community/F5A8C5F3-5B0D-449A-9FA7-6D44DBCC103F%40krugs.de.
--
You received this message because you are subscribed to the Google Groups "OpenAlex Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openalex-commun...@googlegroups.com.
Kyle Demes
Nov 13, 2025, 9:46:05 AM11/13/25
to Samuel Mok, Rainer M Krug, OpenAlex Support, OpenAlex Community
Thank you for pointing that out Samuel. Rainer, I have terminated that key and issued you a new one (in your inbox now).
Great reminder folks that it is not difficult for us to terminate an exposed key but it is an issue for all users if premium keys get shared. If you see someone else's key or you realize you inadvertently exposed your api_key somewhere, no worries just please let us know (sup...@openalex.org) asap so we can get you a new one.
Thanks for the great discussions everyone and keep 'em coming!
-kyle
---The information in this e-mail, and those ensuing, is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, please destroy this message and notify the sender immediately.
Rainer Krug
Senior Data Specialist
SIB | Swiss Institute of Bioinformatics
CH - Switzerland
Mobile +41 78 630 66 57
Raine...@sib.swiss
www.sib.swiss
--
You received this message because you are subscribed to the Google Groups "OpenAlex Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openalex-commun...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/openalex-community/F5A8C5F3-5B0D-449A-9FA7-6D44DBCC103F%40krugs.de.
--
You received this message because you are subscribed to the Google Groups "OpenAlex Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openalex-commun...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages