MESSAGE
| DATE | 2020-09-14 |
| FROM | From: "Patrick Mevzek"
|
| SUBJECT | Re: [Hangout - NYLXS] cache a object in modperl
|
On Sun, Sep 13, 2020, at 21:51, Wesley Peng wrote:
> For work requirement, I need to access IANA TLD database.
>
> So I wrote this perl module:
> https://metacpan.org/pod/Net::IANA::TLD
>
> But, for each new() in the module, the database file will be downloaded
> from IANA's website.
>
> I know this is pretty Inefficient.
Not only inefficient but you abuse remote resources and you risk having
your access being rate limited or just blocked.
You should use caching features available by HTTP as the resource has an ETag:
$ wget -SqO /dev/null http://www.internic.net/domain/root.zone
HTTP/1.1 200 OK
Date: Mon, 14 Sep 2020 15:17:50 GMT
Server: Apache
Last-Modified: Mon, 14 Sep 2020 05:44:00 GMT
Content-Length: 2164237
Vary: Accept-Encoding
ETag: "21060d-5af3f856f0800"
Accept-Ranges: bytes
Cache-Control: max-age=420
Expires: Mon, 14 Sep 2020 15:22:04 GMT
X-Frame-Options: SAMEORIGIN
Referrer-Policy: origin-when-cross-origin
Content-Security-Policy: upgrade-insecure-requests
Age: 165
Keep-Alive: timeout=2, max=358
Connection: Keep-Alive
Content-Type: text/plain; charset=UTF-8
Content-Language: en
So you can do a conditional GET as long as you store the latest ETag
on your side:
$ wget -SqO /dev/null --header 'If-None-Match: "21060d-5af3f856f0800"' http://www.internic.net/domain/root.zone
HTTP/1.1 304 Not Modified
Date: Mon, 14 Sep 2020 15:20:43 GMT
Server: Apache
Connection: Keep-Alive
Keep-Alive: timeout=2, max=358
ETag: "21060d-5af3f856f0800"
Expires: Mon, 14 Sep 2020 15:22:04 GMT
Cache-Control: max-age=420
Vary: Accept-Encoding
All of this has nothing to do with modperl and very lightly to do with Perl at all in fact.
See also the "Cache-Control" and "Age" headers.
Your module on CPAN should take care of that automatically.
PS: TLDs do not vary so much, fetching once per day or once per week should be enough (with manual exceptional override for those cases that need it). But it depends why you do it. Note that the whole content is also available as a zone transfer from various root servers.
--
Patrick Mevzek
_______________________________________________
Hangout mailing list
Hangout-at-nylxs.com
http://lists.mrbrklyn.com/mailman/listinfo/hangout
|
|
