a
/
gosora
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,267 Commits 2 Branches 2 Tags 63 MiB
Commit Graph

225 Commits

Author SHA1 Message Date
Azareal 69f6b9c9c5 refactor rank precedence with precedence struct in router gen 
add precedence tests
remove dead code from router gen
make more strides towards using writers in router gen
optimise dump request
 2021-03-18 15:10:20 +10:00
Azareal b9d94d0888 hoist runnables shared among all routes in a route group to the route group to reduce code duplication 
rewrite router gen to make indentation maintenance easier
 2021-03-18 10:22:28 +10:00
Azareal 75561508c8 don't log referer when DNT header is set 
skip Cookie headers in req logs
 2021-03-02 18:47:36 +10:00
Azareal 4126e8ed0c add qwant, xenforo, twingly, linkfluence, and new toutiao user agent. 
add Users.CountSearch test cases.
add Users.SearchOffset test cases.
add more Users.Count test cases.
add more Users.BulkGetByName test cases.
reduce boilerplate in other tests.
fix bugs in Users.CountSearch and Users.BulkGetByName
 2021-03-02 17:22:32 +10:00
Azareal e49a79014b Log suspicious requests in a seperate log file. 
Optimise the request logger.
Avoid converting from byte slice to string in router gen.
 2021-02-27 16:52:51 +10:00
Azareal de2221f243 fix main.css not being served on /topics/ 
panic when we try to reference an agent which doesn't exist in gen router init
 2021-02-18 09:37:20 +10:00
Azareal 1f0713bb27 add wappalyzer, yacy user agents 
fix mj12bot user agent parsing
numbers should now work in user agent marks
 2021-02-17 15:27:56 +10:00
Azareal 0740b3bf14 Reduce user agent mark boilerplate in router gen. 
Fix PetalBot detection.
 2021-02-17 10:07:11 +10:00
Azareal 74e0ce492d Weekly Views. 
Add avif mime extension.
Optimise skipping watched files which haven't been changed.
Block zgrab to save resources.
Reduce text and boilerplate.

Add route_topic_list_weekviews_start hook.

Add topic_list.week_views_filter phrase.
Changed footer_powered_by phrase.

You will need to run the updater / patcher for this commit.
 2021-01-03 07:46:24 +10:00
Azareal 75d1d6fcbe add telegram, domcop, maui and alt aspiegel user agents 
reduce boilerplate in routergen
 2020-05-29 08:41:19 +10:00
Azareal d0bc74f58f add cocolyze, babbar, surdotly, awesome_bot user agents 
reduce matching in the router slightly
use hookgen for router_after_filters
use hookgen for router_pre_route
shorten ua buffer variable name
skip more symbols in initial ua loop
shorten vulnscan string
avoid unneccessary fork in ua logic
 2020-05-27 07:08:13 +10:00
Azareal 584a50b461 reuse gzip writers with a sync pool 2020-04-13 14:54:47 +10:00
Azareal faf215f388 try speeding up static files with brotli 
track ref routes for a bit to debug them
hit more bots with MicroNotFound
 2020-04-13 13:28:52 +10:00
Azareal 1dc69ed89e add headless chrome ua 2020-04-12 18:33:24 +10:00
Azareal 7d30a66d26 try to filter out bot referrers more effectively 
megaindex update didn't save properly
 2020-04-09 19:30:51 +10:00
Azareal 49a3837621 add netcraft user agent and alt mj12bot matcher 2020-04-04 19:03:11 +10:00
Azareal 4d45fb8710 add burf.co user agent 2020-04-04 13:21:59 +10:00
Azareal afb05ab409 add nimbostratus user agent 2020-04-04 13:10:18 +10:00
Azareal 21d09db477 add CCBot user agent 2020-03-31 23:10:29 +10:00
Azareal fc6544d620 add mattermost ua 
save bytes in req dump

phrases:

rename panel_group_promotion_registered_for to panel_group_promotion_reg_for
rename panel_group_promotion_registered_months_suffix to panel_group_promotion_reg_months_suffix
rename panel_group_promotion_registered_days_suffix to panel_group_promotion_reg_days_suffix
rename panel_group_promotion_registered_hours_suffix to panel_group_promotion_reg_hours_suffix
 2020-03-28 15:39:05 +10:00
Azareal b53577a126 fix MicroNotFound 
reduce allocs in cdn push
 2020-03-27 22:00:29 +10:00
Azareal 3adee57b37 save allocs in router 
better logging for bad char in path
router_pre_route hook now returns properly
 2020-03-27 16:05:35 +10:00
Azareal a668cd296b save bytes in the client templates 
rename template file and functions to reduce bandwidth usage
replace x-loggedin with theoretically faster x-mem
remove redundant check in ua loop
move extraData initialisation down to where it's needed
 2020-03-23 09:14:08 +10:00
Azareal 6b7c51a604 eliminate extraData related allocs in /s/ hot path 
shorten generated simple bot bits
 2020-03-22 16:24:31 +10:00
Azareal ade8f82af7 add python and go user agents. 
add new bot class for python, go and curl to reduce resource usage.
don't push canonicals for loggedin users
try to avoid pushing google site verify when it isn't necessary
give ahrefs the semrush treatment
 2020-03-22 14:37:45 +10:00
Azareal 22ec566a2c add website datenbank user agent 
reduce redundancy and fix sublties in StaticFile()
 2020-03-21 17:44:33 +10:00
Azareal 6a7b8fd3ee map msnbot to bing and avoid bytes for ahrefs too 2020-03-19 07:21:06 +10:00
Azareal b66a494f1c Revert "Revert "will a pointer make this faster...?"" 
This reverts commit 501bcc2425.
 2020-03-18 19:21:34 +10:00
Azareal 501bcc2425 Revert "will a pointer make this faster...?" 
This reverts commit 8e414486b8.
 2020-03-15 21:34:57 +10:00
Azareal 8e414486b8 will a pointer make this faster...? 2020-03-15 17:06:44 +10:00
Azareal 8d19048c0d reduce bytes used on semrush 2020-03-14 12:09:18 +10:00
Azareal bfea6b740f add cliqz, blexbot, curl user agents 
add a /static/ redirect for the slower bots x.x
tweak slack, facebook ua phrases
avoid more allocs for ajax topic
 2020-03-13 16:37:18 +10:00
Azareal aa32bd335c try to avoid as many upfront gzip allocs to avoid wasting resources when they're not needed 
avoid gzip for guests on routeAPI
 2020-03-12 16:52:26 +10:00
Azareal 64ec820453 add MojeekBot user agent 2020-03-11 18:54:11 +10:00
Azareal d43bffdec5 experiment with ajax load for topic 
track route perf for BadRoute
 2020-03-11 13:26:33 +10:00
Azareal d586220df2 add alternate duckduckgo ua 2020-03-09 20:49:22 +10:00
Azareal 28f210371e try to optimise routeAPI 2020-03-09 15:16:44 +10:00
Azareal f5c9a34fdd simple top forum topiclist caching 
reduce thaw period to 4
fix the _ ua bug
try using a hard period for routeAPI to avoid unneccessary churn for a return string that is unlikely to change
 2020-03-08 18:04:37 +10:00
Azareal 6387e2bce0 fix safari ua matcher 2020-03-08 14:36:27 +10:00
Azareal 7d2be466b3 optimise ua tracking 
reduce number of bytes used when we don't know the lang
add more spammy domain strings
 2020-03-08 14:15:27 +10:00
Azareal f55a7a23ba add haosou, proximic and mail_ru user agents 
add pre_global js hook
try to fix archive.org ua patching
don't word split uas at _
 2020-03-08 10:50:23 +10:00
Azareal d22d6f0b27 add AhrefsBot user agent 2020-03-07 21:08:25 +10:00
Azareal 998cbbfd50 add MJ12bot user agent 
add toutiao lang string
 2020-03-07 20:46:27 +10:00
Azareal a80c52861b add toutiao and alternate slack user agents 
track route perf for avatars, robots.txt and opensearch
 2020-03-07 15:28:25 +10:00
Azareal 46b23ee3a9 log unknown uas 2020-03-07 14:20:42 +10:00
Azareal b1014d5647 add archive.org user agent 2020-03-07 13:49:44 +10:00
Azareal ce306d2a4b add slurp, exabot, sogou and alexa user agents 
try again to add aspiegelbot
 2020-03-07 13:25:50 +10:00
Azareal 24eb9f287c try to add AspiegelBot user agent 
tweaked zgrab phrase
 2020-03-06 19:12:41 +10:00
Azareal d2828471cd optimise www redirect check 
minor init.js optimisation
avoid gzip for empty obj in alerts endpoint
 2020-03-06 13:07:28 +10:00
Azareal c31ca03a0d use string builder in buildFlexiWhere 
add string builder growth hints
track favicon perf
 2020-03-05 09:56:45 +10:00