marty/rm-circle-team-router
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Watch #220, and say who actually caught the money

Browse Source 
Marty asked for a watchdog on position #220: tell him when they buy their
upgrade and whether Orlando's #30 gets paid.

The chain says it should. #220 sits at Apex with four uplines between them
and #30, and a level-6 upgrade skips exactly those four, so #30 is the first
candidate the contract tests and it passes both gates (level 5 > 4, three
directs). That is a prediction, though, and the watcher does not report
predictions: it reads MemberUpgraded and UplineRewarded off the logs and says
who was actually paid and how much. If somebody else catches it, it says so.

Tested by replaying real history (BACK=600 over a live #319 upgrade) on both
a wide-range endpoint and a chunked 50-block one, then the dedupe guard, then
two outage cases.

The outage cases mattered. A watchdog that dies quietly is worse than no
watchdog, because silence gets read as "nothing happened", so:

  - a failed scan never advances the block pointer; unread blocks are read
    on the next run rather than skipped
  - four consecutive failures sends a warning instead of going quiet, and
    both failure paths feed one counter. The first version only counted the
    log scan, so a total outage -- where even the block number is unreachable
    -- raised straight past the counter, and the one outage most worth
    shouting about was the one that would have stayed silent.

Also brings rmc-depth-watch.py into the repo. Both watchers existed only on
core, with no copy anywhere, so a rebuilt box lost them silently. The README
records the restore steps and which Polygon RPCs actually serve eth_getLogs
(publicnode and tenderly take 2000 blocks; drpc and 1rpc cap at 50; four
others refuse outright), measured from core rather than assumed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
martbost
2026-09-24 16:25:27 -05:00
parent 1555d4ef6f
commit 09e9d469dc
5 changed files with 494 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/server/README.md
+91
View File
@@ -0,0 +1,91 @@
# Chain watchers (they run on core, not in the app)
Two standalone scripts that read the RM Circle contract and message Marty's Hermes chat.
They are deliberately **not** part of the site: they need no database, no config volume and
no deploy, and they must keep working even when the app is down. They live at `/root/` on
core (`ssh root@coolify.saasy.top`) with their cron entries in `/etc/cron.d/`. Copies are
kept here so a rebuilt box can be put straight back.
| file | on the server |
|---|---|
| `rmc-depth-watch.py` | `/root/rmc-depth-watch.py` |
| `cron.d-rmc-depth-watch` | `/etc/cron.d/rmc-depth-watch` |
| `rmc-member-watch.py` | `/root/rmc-member-watch.py` |
| `cron.d-rmc-member-watch` | `/etc/cron.d/rmc-member-watch` |
Restoring one:
```sh
scp tools/server/rmc-member-watch.py root@coolify.saasy.top:/root/rmc-member-watch.py
ssh root@coolify.saasy.top 'chmod +x /root/rmc-member-watch.py'
scp tools/server/cron.d-rmc-member-watch root@coolify.saasy.top:/etc/cron.d/rmc-member-watch
ssh root@coolify.saasy.top 'chmod 644 /etc/cron.d/rmc-member-watch'
```
Both read the Telegram token from `/root/.mbhermes-telegram-token` (never in this repo) and
keep their own state file under `/root/.rmc-*.json`. Debian cron **ignores `CRON_TZ`**, so
every schedule in those files is UTC.
## rmc-depth-watch.py
Warns when an organisation grows deeper than its apex's level can collect from. A member at
depth *d* pays the apex on their upgrade to level *d+1*, and only when `apex.level >= d`.
Upgrades stop at level 8, so `levelIndex` never exceeds 6 and **depth 7 is a hard ceiling** —
nothing deeper pays an apex at any level, so that is not a miss and must never alert.
Speaks only when the verdict changes, so a steady state stays silent.
```sh
APEX=1154 python3 rmc-depth-watch.py # normal
DRY=1 APEX=21 python3 rmc-depth-watch.py # print, never send, never write state
```
## rmc-member-watch.py
Watches one position and reports every upgrade it makes together with who caught the
pass-up payment and how much. Added 24 September 2026 for `WATCH=220`: #220 sits at Apex,
and a Fastigium upgrade skips the four uplines between them and #30, so Orlando's #30
should catch 9,942.44 POL. The message reports **what the contract did**, read off the
logs — if somebody else catches it, that is what it says.
```sh
WATCH=220 python3 rmc-member-watch.py # normal (cron, every 10 min)
DRY=1 WATCH=220 python3 rmc-member-watch.py # print only, no send, no state write
DRY=1 WATCH=319 BACK=600 python3 rmc-member-watch.py # rehearse against real history
```
`BACK` is how it was tested before going live: replayed over a real #319 upgrade, on both a
wide-range endpoint and a chunked 50-block one.
## Two rules both scripts follow
1. **Never advance the block pointer on a failed scan.** A watchdog that quietly skips the
blocks it could not read is worse than none: the event passes, nothing is said, and
silence gets read as "nothing happened".
2. **Going blind is itself news.** Four consecutive failures sends a warning. Both failure
paths — the block number and the log scan — feed one counter, because an earlier version
counted only the log scan, and a total outage raised straight past it: the single outage
most worth shouting about was the one that would have stayed silent.
## Polygon RPCs, measured from core rather than assumed
| endpoint | `eth_getLogs` |
|---|---|
| `polygon-bor-rpc.publicnode.com` | 2000-block span |
| `polygon.gateway.tenderly.co` | 2000-block span |
| `polygon.drpc.org` | **50 blocks max** |
| `1rpc.io/matic` | **50 blocks max** |
| polygon-rpc.com, ankr, blastapi, blockpi | refuse (401 / 401 / 403 / 521) |
Any client must chunk to the endpoint's own limit. **publicnode rate-limits core's IP
routinely**, because the site already polls it from the same address, so a 429 there is
ordinary rather than a fault — the watcher walks the whole endpoint list twice, 20 seconds
apart, before it will call a run failed. All of these also need a `User-Agent` header:
publicnode 403s the default Python-urllib one.
Event shapes, verified against live logs:
```
MemberUpgraded(uint48 id, uint8 newLevel, ...) topics[1]=id, data[0]=new level
UplineRewarded(uint48 to, uint48 from, uint8, uint256) topics[1]=to, topics[2]=from,
data[0]=level, data[1]=amount wei
```
tools/server/cron.d-rmc-depth-watch
+9
View File
@@ -0,0 +1,9 @@
# RM Circle: ping Marty when an organisation outgrows what its apex level can collect from.
# A member at depth d pays the apex on their upgrade to level d+1, only if apex.level >= d.
# Depth 7 is a hard ceiling (levelIndex tops out at 6), so deeper is not a miss.
# NOTE: Debian cron IGNORES CRON_TZ. Schedule in UTC. 09:15 UTC = 4:15 AM CDT.
# Only speaks when the verdict changes, so a steady state stays silent.
0 */2 * * * root APEX=1154 /usr/bin/python3 /root/rmc-depth-watch.py >> /var/log/rmc-depth-watch.log 2>&1
15 9 * * * root APEX=139 /usr/bin/python3 /root/rmc-depth-watch.py >> /var/log/rmc-depth-watch.log 2>&1
20 9 * * * root APEX=137 /usr/bin/python3 /root/rmc-depth-watch.py >> /var/log/rmc-depth-watch.log 2>&1
25 9 * * * root APEX=21 /usr/bin/python3 /root/rmc-depth-watch.py >> /var/log/rmc-depth-watch.log 2>&1
tools/server/cron.d-rmc-member-watch
+7
View File
@@ -0,0 +1,7 @@
# RM Circle: watch position #220 and say who catches the money when they upgrade.
# Marty asked for this on 24 Sep 2026: #220 sits at Apex, and a level-6 (Fastigium) upgrade
# skips the four uplines between them and #30, so Orlando should catch 9,942.44 POL. The
# watcher reports what the CONTRACT DID, not what that walk predicted.
# Every 10 min: ~285 Polygon blocks per run, two RPC calls. Silent unless #220 moves.
# It never advances its block pointer on a failed scan, and says so if it goes blind.
*/10 * * * * root WATCH=220 /usr/bin/python3 /root/rmc-member-watch.py >> /var/log/rmc-member-watch.log 2>&1
tools/server/rmc-depth-watch.py
+146
View File
@@ -0,0 +1,146 @@
#!/usr/bin/env python3
"""RM Circle: warn when an organisation grows deeper than its apex can collect from.
The rule is read off the contract's own _payUpline, not inferred: a member at depth d under
the apex pays the apex on their upgrade to level d+1, and only when apex.level >= d.
Upgrades stop at level 8, so levelIndex never exceeds 6 and DEPTH 7 IS A HARD CEILING --
nothing deeper than 7 pays the apex at any level, so that is not a miss and must not alert.
Pings Marty's phone only when the verdict changes, so a steady state stays silent.
APEX=1154 python3 depth_watch.py # normal
DRY=1 APEX=21 python3 depth_watch.py # print, never send, never write state
"""
import json
import os
import pathlib
import sys
import urllib.request
CONTRACT = "0x33bdaeefd6d17d80ae53816c916dfb26c4fb2daf"
RPCS = ["https://polygon-bor-rpc.publicnode.com", "https://1rpc.io/matic"]
APEX = int(os.environ.get("APEX", "1154"))
DRY = os.environ.get("DRY") == "1"
CEILING = 7 # levelIndex tops out at 6, so depth 7 is the last that can pay an apex
MAX_WALK = 16 # MAX_MATRIX_DEPTH in the contract
STATE = pathlib.Path(os.environ.get("STATE_FILE", f"/root/.rmc-depth-watch-{APEX}.json"))
TOKEN_FILE = pathlib.Path(os.environ.get("TG_TOKEN_FILE", "/root/.mbhermes-telegram-token"))
CHAT_ID = os.environ.get("TG_CHAT", "1289244227")
def call(data):
body = json.dumps({"jsonrpc": "2.0", "id": 1, "method": "eth_call",
"params": [{"to": CONTRACT, "data": data}, "latest"]}).encode()
last = None
for url in RPCS:
try:
# publicnode 403s the default Python-urllib agent; the Node client gets through
# because it sends one at all, so send a plain identifiable agent of our own.
req = urllib.request.Request(url, data=body, headers={
"Content-Type": "application/json", "User-Agent": "rmc-depth-watch/1.0"})
out = json.load(urllib.request.urlopen(req, timeout=30))
if "error" in out:
raise RuntimeError(out["error"].get("message", "rpc error"))
return out["result"]
except Exception as exc: # try the next endpoint before giving up
last = exc
raise last
def enc(n):
return format(int(n), "064x")
def word(res, i):
return int(res[2 + i * 64: 2 + (i + 1) * 64], 16)
def member(mid):
r = call("0xc92463fa" + enc(mid))
return {"id": mid, "level": word(r, 5), "directCount": word(r, 6)}
def children(mid):
"""getMatrixChildren returns a FIXED-size array, inlined: read the words straight off.
Decoding it as a dynamic array (offset + length) yields garbage."""
r = call("0x04c8cc3d" + enc(mid))
n = (len(r) - 2) // 64
return [w for w in (word(r, i) for i in range(n)) if w > 0]
def send(text):
token = TOKEN_FILE.read_text().strip()
body = json.dumps({"chat_id": CHAT_ID, "text": text}).encode()
req = urllib.request.Request(f"https://api.telegram.org/bot{token}/sendMessage",
data=body, headers={"Content-Type": "application/json"})
urllib.request.urlopen(req, timeout=30).read()
def main():
# breadth-first: one call per member, not one per path
depth, frontier, seen, count = 0, [APEX], {APEX}, 0
while frontier and depth < MAX_WALK:
nxt = []
for mid in frontier:
for c in children(mid):
if c not in seen:
seen.add(c)
nxt.append(c)
count += 1
if not nxt:
break
frontier, depth = nxt, depth + 1
apex = member(APEX)
covered = min(apex["level"], CEILING)
reachable = min(depth, CEILING)
missing = list(range(covered + 1, reachable + 1))
behind = bool(missing)
at_edge = (not behind) and reachable == covered and covered < CEILING and depth > 0
nxt_rung = apex["level"] + 1
line = f'#{APEX} level {apex["level"]}, {count} in the org, {depth} deep'
msg = None
if behind:
listed = (", ".join(str(d) for d in missing[:-1]) + " and " + str(missing[-1])
if len(missing) > 1 else str(missing[0]))
msg = (f"RM Circle: #{APEX} is missing generations.\n{line}.\n"
f"Depth {listed} upgrade{'s' if len(missing) > 1 else ''} route past you. "
f"Level {nxt_rung} picks up depth {covered + 1}"
+ (", and each rung after that adds one more generation." if len(missing) > 1 else ".")
+ ("\nDepth 8 and below never pays an apex at any level, so that part is not a miss."
if depth > CEILING else ""))
elif at_edge:
msg = (f"RM Circle: #{APEX} is at its edge.\n{line}.\n"
f"One more generation and that depth's upgrade routes past you. "
f"Level {nxt_rung} covers it.")
key = f"{reachable}:{covered}:{'behind' if behind else 'edge' if at_edge else 'ok'}"
prev = {}
try:
prev = json.loads(STATE.read_text())
except Exception:
pass
changed = prev.get("key") != key
verdict = ("missing depth " + ", ".join(str(d) for d in missing)) if behind else \
("at the edge" if at_edge else "covered")
print(f"{line} -> {verdict} ({'changed' if changed else 'unchanged'})", flush=True)
if msg and changed and not DRY:
send(msg)
print("telegram: sent", flush=True)
elif msg:
print(("would send" if DRY else "nothing new to say") + ":\n" + msg, flush=True)
if not DRY:
STATE.write_text(json.dumps({"key": key, "depth": depth, "level": apex["level"],
"count": count}))
if __name__ == "__main__":
try:
main()
except Exception as exc:
print(f"depth_watch failed: {exc}", file=sys.stderr)
sys.exit(1)
tools/server/rmc-member-watch.py
+241
View File
@@ -0,0 +1,241 @@
#!/usr/bin/env python3
"""RM Circle: tell Marty the moment a watched position upgrades, and who caught the money.
Built for one question he asked on 24 September 2026: when #220 buys Fastigium, does Orlando's
#30 actually get paid? The chain said it should -- #220 sits at Apex with four uplines between
them and #30, and a level-6 upgrade skips exactly those four -- but "should" is not "did", so
this reports what the CONTRACT DID, never what the walk predicted. If somebody else catches it,
that is what the message says.
MemberUpgraded(uint48 id, uint8 newLevel, ...) topics[1]=id, data[0]=new level
UplineRewarded(uint48 to, uint48 from, uint8, uint256) topics[1]=to, topics[2]=from,
data[0]=level, data[1]=amount wei
Two rules this thing lives by:
1. It NEVER advances its block pointer on a failed scan. A watchdog that quietly skips the
blocks it could not read is worse than no watchdog: the event passes, nothing is said, and
everyone assumes silence means nothing happened.
2. If it cannot reach the chain several runs running, it SAYS SO. Going blind is itself news.
WATCH=220 python3 rmc-member-watch.py # normal (cron)
DRY=1 WATCH=220 python3 rmc-member-watch.py # print only, no send, no state write
BACK=50000 DRY=1 WATCH=139 python3 ... # rehearse on history instead of the pointer
"""
import json
import os
import pathlib
import sys
import time
import urllib.request
CONTRACT = "0x33bdaeefd6d17d80ae53816c916dfb26c4fb2daf"
# (url, the widest block span that endpoint accepts in one eth_getLogs). All four were measured
# from core itself, not assumed: polygon-rpc.com, ankr, blastapi and blockpi all refuse it outright.
# The order matters -- publicnode and tenderly take the whole window in one request, so a normal
# run costs two calls; the 50-block pair are chunked fallbacks for when those rate-limit. They do:
# the RM Circle app already polls publicnode from this same IP, so 429s here are routine, not alarming.
RPCS = [("https://polygon-bor-rpc.publicnode.com", 2000),
("https://polygon.gateway.tenderly.co", 2000),
("https://polygon.drpc.org", 50),
("https://1rpc.io/matic", 50)]
ROUNDS = 2 # a whole pass over every endpoint, twice, before calling a run failed
ROUND_WAIT = 20 # seconds between those passes, to let a rate limit lapse
T_UPGRADED = "0xc0b79a9e133d4dcbb1a606a57591d98dce93c7d5c86197a5caae22c4a1480049"
T_UPLINE = "0x6cdacf6757bdea1fcf01918794f79ca1e71b088fbec8fde7b027a963cfab0ae9"
LEVELS = ['Scintilla', 'Ascensus', 'Fabrica', 'Culmen', 'Apex', 'Fastigium', 'Vertex', 'Corona']
NAMED = {30: 'Orlando'}
WATCH = int(os.environ.get("WATCH", "220"))
DRY = os.environ.get("DRY") == "1"
BACK = int(os.environ.get("BACK", "0"))
STATE = pathlib.Path(os.environ.get("STATE_FILE", "/root/.rmc-member-watch-%d.json" % WATCH))
TOKEN_FILE = pathlib.Path(os.environ.get("TG_TOKEN_FILE", "/root/.mbhermes-telegram-token"))
CHAT_ID = os.environ.get("TG_CHAT", "1289244227")
WARN = "\u26a0\ufe0f"
BLIND_AFTER = 4 # consecutive failed runs before we admit we have gone blind
def rpc(url, method, params, timeout=45):
body = json.dumps({"jsonrpc": "2.0", "id": 1, "method": method, "params": params}).encode()
# publicnode 403s the default Python-urllib agent, so always send one of our own
req = urllib.request.Request(url, data=body, headers={
"Content-Type": "application/json", "User-Agent": "rmc-member-watch/1.0"})
out = json.load(urllib.request.urlopen(req, timeout=timeout))
if "error" in out:
raise RuntimeError(out["error"].get("message", "rpc error"))
return out["result"]
def head():
last = None
for attempt in range(ROUNDS):
for url, _ in RPCS:
try:
return int(rpc(url, "eth_blockNumber", []), 16), url
except Exception as exc:
last = exc
if attempt + 1 < ROUNDS:
time.sleep(ROUND_WAIT)
raise last
def get_logs(lo, hi):
"""Every log for the watched member between two blocks, from whichever endpoint answers.
Chunked to each endpoint's own range limit, so a 50-block-max fallback still works."""
pad = "0x" + format(WATCH, "064x")
filters = [
{"topics": [T_UPGRADED, pad]}, # the watched member upgrading
{"topics": [T_UPLINE, None, pad]}, # what that upgrade paid, and to whom
]
last = None
for attempt in range(ROUNDS):
for url, span in RPCS:
try:
out = []
for f in filters:
start = lo
while start <= hi:
end = min(start + span - 1, hi)
q = dict(f)
q.update({"address": CONTRACT, "fromBlock": hex(start), "toBlock": hex(end)})
out += rpc(url, "eth_getLogs", [q])
start = end + 1
return out, url
except Exception as exc:
last = exc # start over on the next endpoint
if attempt + 1 < ROUNDS:
time.sleep(ROUND_WAIT) # every endpoint balked; give the limits a rest
raise last
def send(text):
token = TOKEN_FILE.read_text().strip()
body = json.dumps({"chat_id": CHAT_ID, "text": text, "parse_mode": "HTML",
"disable_web_page_preview": True}).encode()
req = urllib.request.Request("https://api.telegram.org/bot%s/sendMessage" % token,
data=body, headers={"Content-Type": "application/json"})
urllib.request.urlopen(req, timeout=30).read()
def lvname(n):
return LEVELS[n - 1] if 1 <= n <= len(LEVELS) else "level %d" % n
def who(mid):
return "#%d (%s)" % (mid, NAMED[mid]) if mid in NAMED else "#%d" % mid
def load():
try:
return json.loads(STATE.read_text())
except Exception:
return {}
def compose(tx, upgrades, pays):
"""One message per transaction: what was bought, and where every POL of it went."""
lines = []
for lv in sorted(upgrades):
lines.append("<b>#%d upgraded to %s</b> (level %d)." % (WATCH, lvname(lv), lv))
if not upgrades:
lines.append("<b>#%d generated a pass-up payment.</b>" % WATCH)
for to, pol in pays:
lines.append("\n%s POL went to %s." % (format(pol, ",.2f"), who(to)))
if not pays:
lines.append("\nNo pass-up payment was logged against it, so it fell through to the "
"admin address: nobody upline qualified.")
lines.append("\nhttps://polygonscan.com/tx/" + tx)
return "\n".join(lines)
def stumble(st, exc, where):
"""One place for every way a run can fail to read the chain.
Whatever broke -- the block number or the log scan -- the pointer stays put and the failure
is COUNTED. An earlier version only counted failures from the log scan, so a total outage
(where even the block number is unreachable) raised straight past the counter: the one kind
of outage worth shouting about was the one that would have stayed silent.
"""
fails = int(st.get("fails", 0)) + 1
print("%s FAILED (%d in a row): %s" % (where, fails, exc), file=sys.stderr)
if fails >= BLIND_AFTER and not st.get("blindAlerted") and not DRY:
try:
send(WARN + " <b>The #%d watchdog cannot reach the chain.</b>\n\n"
"%d runs in a row have failed, so nothing past block %s has been read and an "
"upgrade could pass unnoticed. Last error: %s"
% (WATCH, fails, st.get("lastBlock", "?"), exc))
st["blindAlerted"] = True
except Exception as e2:
# Telegram is unreachable too; log it and let the next run try again
print("could not send the blind alert: %s" % e2, file=sys.stderr)
st["fails"] = fails
if not DRY:
STATE.write_text(json.dumps(st))
sys.exit(1)
def main():
st = load()
try:
bn, _ = head()
except Exception as exc:
stumble(st, exc, "block number")
if BACK:
lo, hi = bn - BACK, bn
elif "lastBlock" in st:
lo, hi = st["lastBlock"] + 1, bn
else:
# first run: a short look back covers the gap between writing this and the first tick
lo, hi = bn - 1000, bn
if lo > hi:
print("nothing new (head %d)" % bn, flush=True)
return
try:
logs, src = get_logs(lo, hi)
except Exception as exc:
stumble(st, exc, "scan %d-%d" % (lo, hi))
# group by transaction so the upgrade and the payment it caused arrive as one message
txs = {}
for l in logs:
tx = l["transactionHash"]
d = txs.setdefault(tx, {"block": int(l["blockNumber"], 16), "upgrades": [], "pays": []})
data = l["data"]
word = lambda i: int(data[2 + i * 64: 2 + (i + 1) * 64], 16)
if l["topics"][0] == T_UPGRADED:
d["upgrades"].append(word(0))
else:
d["pays"].append((int(l["topics"][1], 16), word(1) / 1e18))
seen = set(st.get("seen", []))
order = sorted(txs.items(), key=lambda kv: kv[1]["block"])
fresh = [(t, d) for t, d in order if t not in seen]
print("scanned %d-%d via %s: %d log(s), %d tx, %d new"
% (lo, hi, src, len(logs), len(txs), len(fresh)), flush=True)
for tx, d in fresh:
msg = compose(tx, d["upgrades"], d["pays"])
if DRY:
print("--- would send ---\n" + msg, flush=True)
else:
send(msg)
print("telegram: sent for " + tx, flush=True)
seen.add(tx)
if not DRY:
st.update({"lastBlock": hi, "seen": sorted(seen)[-200:], "fails": 0,
"blindAlerted": False, "at": int(time.time())})
STATE.write_text(json.dumps(st))
if __name__ == "__main__":
try:
main()
except SystemExit:
raise
except Exception as exc:
print("member_watch failed: %s" % exc, file=sys.stderr)
sys.exit(1)