Show HN: Open-source script to get your site indexed on Google
131 points by goenning 1 year ago | 61 comments- xnx 1 year agoThis script abuses the Indexing API which is intended for job posting and other specific purposes. https://developers.google.com/search/apis/indexing-api/v3/qu...
Use at your own risk.
- ldoughty 1 year agoMight simply not work for your website:
> Currently, the Indexing API can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject.
I wanted to highlight (in addition to your statement) that JobPosting is a specific type of structured data.
If the target site doesn't have these elements, it may or may not work... or it may work for now, but not once they realized it's being used incorrectly
JobPosting structured data: https://developers.google.com/search/docs/appearance/structu...
- datashaman 1 year ago> If you dont have content that falls into those categories, the API won't help you there.
https://www.youtube.com/watch?v=kvYb2bdtT7A&t=422s
IME it will silently drop and ignore anything that is not relevant.
- ldoughty 1 year ago
- adaboese 1 year agoThe annoying thing about this is that it will ruin this feature for everyone else. I, and many others, use this for requesting to index time sensitive content.
- chiefalchemist 1 year agoYes and no. I mean, just because something gets indexed doesn't mean Google values it and is willing to expose its customers to it.
The consistent problem with SEO is that most SEOs don't understand Google's business model. They don't understand Google is going to best serve its customers (i.e., those doing the search). SEOs (and their clients) need to understand that getting Google to index a turd isn't going to change the fact that the content and the experience i'ts wrapped in is still a turd. Google is not interested in pointing its customers to turds.
- hammyhavoc 1 year agoFor a company not interested in pointing their customers to turds, they sure do point them to a lot of turds.
- chiefalchemist 1 year agoThat's not what it wants to do. Yes, that is what's happening, for a number of reasons. Without people searching, there are no eye-balls. Put another way, the sites being indexed and ranked are *not* the customer(s).
- chiefalchemist 1 year ago
- solumunus 1 year ago> Google is not interested in pointing its customers to turds.
We must have been using a different Google over the past 3 years. It does this almost exclusively now.
- water-your-self 1 year agoGoogle is not interested in pointing its customers to a turd that hasn't paid for that right.
- water-your-self 1 year ago
- dindobre 1 year agoI'm not sure I agree that the people doing the search are the customers here
- Sai_ 1 year agoI’m really curious to know who exactly you think are Google Search’s customers in the context of this thread about SEO.
- Sai_ 1 year ago
- cynicalsecurity 1 year agoSEO died many years ago, but some companies are still trying to sell their naive clients some magical "SEO optimisation". Which is plainly scam at this point.
- adaboese 1 year agoThere are a ton of SEO optimizations that are extremely significant:
* performance/SSR * interlinking/dead links * keyword cannibalization
to name a few
- adaboese 1 year ago
- jknoepfler 1 year agoEyeballs are not Google's customers, paying advertisers are Google's customers.
If a paying customer gives Google money to point eyeballs to turds, it points eyeballs to turds (this is how Google makes money today, it is the business model for search). The problem with SEO isn't that it degrades search, it's that SEO users aren't paying customers and don't make Google any money (and compromises Google's ability to direct eyeballs to paying customers).
This is classical "enshitification" - offer a service for free to capture eyeball share, then offer a paid service to companies that capitalizes on that eyeball share but compromises the "eyeball experience" (and then in the endgame, squeeze companies that become dependent upon the eyeball-platform to serve shareholders).
- hammyhavoc 1 year ago
- chiefalchemist 1 year ago
- AznHisoka 1 year agoAnother easy way is to just tweet it, which works for me - they usually get indexed < 1 hour later. Google has access to tweets and the URLs in those tweets.
- gasparto 1 year agoWhat happend to the good'ol sitemap.xml?
You'll probably find an npm package with lots of dependencies that'll generate that sitemap for you if that's what you need...
- sjwhevvvvvsj 1 year agoI’m failing to see how this isn’t just “hey look at my sitemap”!
- wut42 1 year agoPer Google's docs:
> For websites with many short-lived pages like job postings or livestream videos, we recommend using the Indexing API instead of sitemaps because the Indexing API prompts Googlebot to crawl your pages sooner than updating the sitemap
- wut42 1 year ago
- sjwhevvvvvsj 1 year ago
- mortallywounded 1 year agoIs this any different from logging into the Google search console and submitting your sitemap/index request?
- rkuykendall-com 1 year agoI submitted 1,900 pages in September and it has yet to look at 600. It did 4 this month.
I wish I had been more picky with my sitemap but I thought including all URLs was the goal. I at least properly weighted them but that doesn't seem to do much.
- 15457345234 1 year agoIf the organic results are too good someone might click on one of them instead of a sponsored link or advert. De-indexing/non-indexing helps to avoid that outcome.
- qingcharles 1 year agoSame here.
- 15457345234 1 year ago
- jklinger410 1 year agoA little reading would get you the answer: https://seogets.com/blog/google-indexing-script
- paradite 1 year agoThe info on 24 to 48 hours waiting time is wrong. I just submitted a few new pages (not yet indexed). They got indexed in less than 4 hours.
- paradite 1 year ago
- tomschwiha 1 year agoSubmitting manually takes sooooo long.
- goenning 1 year agosame result from what I've seen, but not scalable for larger amount of pages
- rkuykendall-com 1 year ago
- dakiol 1 year agoIs there something for the opposite? I don't want google (or any other scrapper) to index my website. Afaik, robots.txt is not authoritative.
- speedgoose 1 year ago
That should do.sudo iptables -A INPUT -p tcp --dport 80 -j DROP sudo iptables -A INPUT -p tcp --dport 443 -j DROP sudo ip6tables -A INPUT -p tcp --dport 80 -j DROP sudo ip6tables -A INPUT -p tcp --dport 443 -j DROP
- speedgoose 1 year ago
- ahmedfromtunis 1 year agoFrom Indexing API documentation:
> Currently, the Indexing API can only be used to crawl pages with either `JobPosting` or `BroadcastEvent` embedded in a `VideoObject`.
So this might come with the risk of seeing the site you want to boost rather penalized by Google.
- goenning 1 year agoThis is not a boost, Index != Ranking
- ahmedfromtunis 1 year agoEither way, this is clearly abusing the API by using it for things that it wasn't intended for.
The only outcome I can see from this is a) contributing to the rise of spam and b) harming people managing sites and apps for which this API is vital.
- ahmedfromtunis 1 year ago
- goenning 1 year ago
- mvkel 1 year agoI recently launched a mini project and was shocked at how difficult and long it took to get any of its pages properly indexed on Google.
It's almost as if Google is actively trying -not- to index anything as a way to reduce spam, by forcing the people who really care to jump through 100 hoops.
A great way for the dark web remains dark.
- lobsterthief 1 year agoIt just takes time. Getting people to link to it by sharing it in other channels will help to shorten the timeframe.
- aarreedd 1 year agoIt just takes time
- callalex 1 year agoHow long is long in your case?
- lobsterthief 1 year ago
- leros 1 year agoI just submit a sitemap URL to Google Search Console Tools. Is this any different?
- ninefoxgambit 1 year agoI’ve seen a lot of indie startups lately that are basically selling faster google indexing then you can get for free using google search console. I guess they are probably using this feature under the hood.
- dewey 1 year agoI've seen some people even wrapping and re-selling this as SaaS.
- nhggfu 1 year ago"to get your site indexed" => a nonsense claim
+ this technique might make engines aware of your content, but doesn't guarantee indexation whatsoever.
- 1 year ago
- cranberryturkey 1 year agoor just submit a sitemap.xml via google console.
- ChrisArchitect 1 year ago? "what I've noticed"...Google only indexing if a site has backlinks or is submitted by owner. Uh..yeah, how else would google know about a new URL? C'mon. This just seems like the usual SEO obsession/grift with some 'secret' way to get things done. It's straightfwd these days. Are you saying none of the pages you're queuing up are linked to each other? Most cases they would be in some way right? So the spider will start indexing them all based on a top url submission or a few key urls. Do event/job board sites really need all of their pages to be indexed immediately?
- navigate8310 1 year agoSo, Google stopped automating indexation because spam, humanity finds new way to resume automation to again propagate spam. It seems Google is trapped in its toxic game of search engine optimization.
- beeboobaa 1 year agoGoogle no longer finds new sites automatically? That might explain why it's been so trash the past few years.
I remember running a few websites back in the day, and with zero interaction with google all of the pages showed up in the search index a day or two after publishing at most.
- nextaccountic 1 year ago
- 1 year ago
- harryp_peng 1 year agolol
- nextaccountic 1 year ago
- goenning 1 year agoThe only possible outcome if for them to shutdown this API or make it work as documented. There's already at least 10+ SaaS offering this as service.
- RobotToaster 1 year agoEh, you can get google to index your site by just submitting a site map, it just takes a little longer.
- mvkel 1 year agoIf "a little longer" is six months+, you're absolutely right.
- mvkel 1 year ago
- 1 year ago
- beeboobaa 1 year ago
- niemal_dev 1 year ago[dead]
- federalauth 1 year agoThank you for sharing. Can you quickly explain why keeping an MD5 avoids the abuse issue?
- niemal_dev 1 year agoYou don't post the same URL twice with the same content therefore you maintain the logic of the API itself, so there's no spam or abuse of any kind.
- niemal_dev 1 year ago
- federalauth 1 year ago