I'm encountering a peculiar issue when trying to make an HTTP GET request to the URL 'https://www.nasdaq.com/feed/rssoutbound?category=FinTech'. When I manually enter this URL in my web browser, the feed loads without any issues. However, when I attempt to make the same request programmatically using code (I've tried C# with HttpClient and Node.js with Axios), the request hangs indefinitely and eventually times out.
Here's my C# code:
public async Task Execute(){ using HttpClient httpClient = new HttpClient(); try { // Specify the URL you want to request string url = "https://www.nasdaq.com/feed/rssoutbound?category=FinTech"; httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("*/*")); httpClient.DefaultRequestHeaders.AcceptLanguage.Add(new StringWithQualityHeaderValue("en-US")); httpClient.DefaultRequestHeaders.AcceptLanguage.Add(new StringWithQualityHeaderValue("en", 0.9)); httpClient.DefaultRequestHeaders.Add("sec-ch-ua", "\"Chromium\";v=\"116\", \"Not)A;Brand\";v=\"24\", \"Microsoft Edge\";v=\"116\""); httpClient.DefaultRequestHeaders.Add("sec-ch-ua-mobile", "?0"); httpClient.DefaultRequestHeaders.Add("sec-ch-ua-platform", "\"Windows\""); httpClient.DefaultRequestHeaders.Add("sec-fetch-dest", "empty"); httpClient.DefaultRequestHeaders.Add("sec-fetch-mode", "cors"); httpClient.DefaultRequestHeaders.Add("sec-fetch-site", "same-origin"); // Add the User-Agent header httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"); // Send a GET request and wait for the response HttpResponseMessage response = await httpClient.GetAsync(url); // Check if the request was successful if (response.IsSuccessStatusCode) { // Read the content of the response as a string string content = await response.Content.ReadAsStringAsync(); // Print the content to the console Console.WriteLine(content); } else { Console.WriteLine($"HTTP request failed with status code: {response.StatusCode}"); } } catch (Exception e) { Console.WriteLine(e); throw; }}I've also tried making the same request with Node.js and Axios, and the result is the same:
// Node.js code using Axiosconst axios = require('axios');async function fetchData() { try { const response = await axios.get('https://www.nasdaq.com/feed/rssoutbound?category=FinTech', { headers: { Accept: '*/*','Accept-Language': 'en-US,en;q=0.9,tr;q=0.8','sec-ch-ua': '"Chromium";v="116", "Not)A;Brand";v="24", "Microsoft Edge";v="116"','sec-ch-ua-mobile': '?0','sec-ch-ua-platform': '"Windows"','sec-fetch-dest': 'empty','sec-fetch-mode': 'cors','sec-fetch-site': 'same-origin','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36' } }); // ... (handling the response) } catch (error) { console.error('An error occurred:', error.message); }}fetchData();Strangely, when I attempt to retrieve feeds from other RSS sources, everything works fine. The issue seems to be specific to the Nasdaq feed. I've even tried using Puppeteer to fetch the content, and it also hangs indefinitely.
I used Edge (browser) tool to send request by clearing all editable headers and without cookie, and it still works. So I doubt it is a cookie problem.
And following PowerShell works as well;
$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession$session.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 Edg/116.0.1938.81"Invoke-WebRequest -UseBasicParsing -Uri "https://www.nasdaq.com/feed/rssoutbound?category=FinTech" `-WebSession $session `-Headers @{"authority"="www.nasdaq.com""method"="GET""path"="/feed/rssoutbound?category=FinTech""scheme"="https""accept"="*/*""accept-encoding"="gzip, deflate, br""accept-language"="en-US,en;q=0.9,tr;q=0.8""sec-ch-ua"="`"Chromium`";v=`"116`", `"Not)A;Brand`";v=`"24`", `"Microsoft Edge`";v=`"116`"""sec-ch-ua-mobile"="?0""sec-ch-ua-platform"="`"Windows`"""sec-fetch-dest"="empty""sec-fetch-mode"="cors""sec-fetch-site"="same-origin"}
What could be causing this problem with the Nasdaq feed specifically? Is there something unique about their server configuration or the way they handle requests that might be causing this behavior? Any insights or suggestions would be greatly appreciated.
