Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 31 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ on:
- main

env:
VERSION_NUMBER: 'v1.10.2'
VERSION_NUMBER: 'v1.10.3'
DOCKERHUB_REGISTRY_NAME: 'digitalghostdev/poke-cli'
AWS_REGION: 'us-west-2'

Expand Down Expand Up @@ -59,10 +59,38 @@ jobs:
with:
sarif_file: results.sarif

bandit:
runs-on: ubuntu-22.04

steps:
- name: Checkout
uses: actions/checkout@v6

- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.12'

- name: Install uv
uses: astral-sh/setup-uv@v7

- name: Run Bandit Security Scanner
run: |
uv tool run --from 'bandit[sarif,toml]' bandit \
-r card_data/pipelines \
-f sarif \
-o bandit-results.sarif \
|| true

- name: Upload SARIF Report
uses: github/codeql-action/upload-sarif@v4
with:
sarif_file: bandit-results.sarif

Comment thread
digitalghost-dev marked this conversation as resolved.
gitleaks:
runs-on: ubuntu-22.04
needs: [gosec]
if: needs.gosec.result == 'success'
needs: [gosec, bandit]
if: needs.gosec.result == 'success' && needs.bandit.result == 'success'

steps:
- name: Checkout
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
name: Python Tests

on:
push:
branches:
- main
paths:
- 'card_data/**'
- 'web/**'
pull_request:
types: [opened, reopened, synchronize]
paths:
Expand Down
2 changes: 1 addition & 1 deletion .goreleaser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ builds:
- windows
- darwin
ldflags:
- -s -w -X main.version=v1.10.2
- -s -w -X main.version=v1.10.3

archives:
- formats: [ 'zip' ]
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# build 1
FROM golang:1.25.9-alpine3.23 AS build
FROM golang:1.25.10-alpine3.23 AS build

WORKDIR /app

Expand All @@ -8,7 +8,7 @@ RUN go mod download

COPY . .

RUN go build -ldflags "-X main.version=v1.10.2" -o poke-cli .
RUN go build -ldflags "-X main.version=v1.10.3" -o poke-cli .

# build 2
FROM --platform=$BUILDPLATFORM alpine:3.23
Expand Down
25 changes: 19 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<img width="425" src="poke-cli.png" alt="pokemon-logo"/>
<h4></h4>
<img src="https://img.shields.io/github/v/release/digitalghost-dev/poke-cli?style=flat-square&logo=git&logoColor=FFCC00&label=Release%20Version&labelColor=EEE&color=FFCC00" alt="version-label">
<img src="https://img.shields.io/docker/image-size/digitalghostdev/poke-cli/v1.10.2?arch=arm64&style=flat-square&logo=docker&logoColor=FFCC00&labelColor=EEE&color=FFCC00" alt="docker-image-size">
<img src="https://img.shields.io/docker/image-size/digitalghostdev/poke-cli/v1.10.3?arch=arm64&style=flat-square&logo=docker&logoColor=FFCC00&labelColor=EEE&color=FFCC00" alt="docker-image-size">
<img src="https://img.shields.io/github/actions/workflow/status/digitalghost-dev/poke-cli/ci.yml?branch=main&style=flat-square&logo=github&logoColor=FFCC00&label=CI&labelColor=EEE&color=FFCC00" alt="ci-status-badge">
</div>
<div align="center">
Expand All @@ -22,6 +22,9 @@ View the [documentation](https://docs.poke-cli.com) on the data infrastructure i
* [Roadmap](#roadmap)
* [Tested Terminals](#tested-terminals)

> ![NOTE]
> A version 2 is being planned and built that will remove/update some commands and flags. Refer to the changes under the [Roadmap](#version-2-changes) for more information.

---

## Demo
Expand Down Expand Up @@ -99,11 +102,11 @@ Cloudsmith is a fully cloud-based service that lets you easily create, store, an
3. Choose how to interact with the container:
* Run a single command and exit:
```bash
docker run --rm -it digitalghostdev/poke-cli:v1.10.2 <command> [subcommand] [flag]
docker run --rm -it digitalghostdev/poke-cli:v1.10.3 <command> [subcommand] [flag]
```
* Enter the container and use its shell:
```bash
docker run --rm -it --name poke-cli --entrypoint /bin/sh digitalghostdev/poke-cli:v1.10.2 -c "cd /app && exec sh"
docker run --rm -it --name poke-cli --entrypoint /bin/sh digitalghostdev/poke-cli:v1.10.3 -c "cd /app && exec sh"
# placed into the /app directory, run the program with './poke-cli'
# example: ./poke-cli ability swift-swim
```
Expand All @@ -112,13 +115,13 @@ Cloudsmith is a fully cloud-based service that lets you easily create, store, an
> The `card` command renders TCG card images using your terminal's graphics protocol. When running inside Docker, pass your terminal's environment variables so image rendering works correctly:
> ```bash
> # Kitty
> docker run --rm -it -e TERM -e KITTY_WINDOW_ID digitalghostdev/poke-cli:v1.10.2 card
> docker run --rm -it -e TERM -e KITTY_WINDOW_ID digitalghostdev/poke-cli:v1.10.3 card
>
> # WezTerm, iTerm2, Ghostty, Konsole, Rio, Tabby
> docker run --rm -it -e TERM -e TERM_PROGRAM digitalghostdev/poke-cli:v1.10.2 card
> docker run --rm -it -e TERM -e TERM_PROGRAM digitalghostdev/poke-cli:v1.10.3 card
>
> # Windows Terminal (Sixel)
> docker run --rm -it -e WT_SESSION digitalghostdev/poke-cli:v1.10.2 card
> docker run --rm -it -e WT_SESSION digitalghostdev/poke-cli:v1.10.3 card
> ```
> If your terminal is not listed above, image rendering is not supported inside Docker.

Expand Down Expand Up @@ -235,6 +238,16 @@ Below is a list of the planned/completed commands and flags:
- [x] `tcg`: get data about TCG tournaments.
- [x] `types`: get data about a specific typing.

### Version 2 Changes
The following planned changes in `v2`:

- `pokemon <name> -t | --types` — removed; typing is included by default.
- `pokemon <name> --defense` - being renamed to `--defenses` to keep consistency with other flags in the `pokemon` command.
- `natures` — moves to a flag under a new `mechanics` command.
- `tcg` — moves to a new `comp` command (covers competitive TCG *and* VGC data).
- Adding `pflag` library to enforce POSIX style flags.


---
## Tested Terminals
| Terminal | OS | Status | Issues |
Expand Down
19 changes: 14 additions & 5 deletions card_data/pipelines/definitions.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@
from .defs.extract.tcgdex.extract_sets import extract_sets_data
from .defs.extract.tcgdex.extract_series import extract_series_data
from .defs.load.limitless.load_standings import load_standings_data
from .defs.load.tcgcsv.load_pricing import load_pricing_data, data_quality_checks_on_pricing
from .defs.load.tcgcsv.load_pricing import (
load_pricing_data,
data_quality_checks_on_pricing,
)
from .defs.load.tcgdex.load_sets import load_sets_data, data_quality_check_on_sets
from .defs.load.tcgdex.load_series import load_series_data, data_quality_check_on_series
from .sensors import discord_success_sensor, discord_failure_sensor
Expand All @@ -18,7 +21,9 @@
@definitions
def defs() -> dg.Definitions:
# load_from_defs_folder discovers dbt assets from transform_data.py
folder_defs: dg.Definitions = load_from_defs_folder(project_root=Path(__file__).parent.parent)
folder_defs: dg.Definitions = load_from_defs_folder(
project_root=Path(__file__).parent.parent
)
return dg.Definitions.merge(
folder_defs,
defs_discord_sensors,
Expand Down Expand Up @@ -56,7 +61,9 @@ def defs() -> dg.Definitions:
# Series pipeline job
series_pipeline = dg.define_asset_job(
name="series_pipeline_job",
selection=dg.AssetSelection.assets(extract_series_data).downstream(include_self=True),
selection=dg.AssetSelection.assets(extract_series_data).downstream(
include_self=True
),
)

defs_series: dg.Definitions = dg.Definitions(
Expand All @@ -78,7 +85,9 @@ def defs() -> dg.Definitions:
# Standings pipeline job
standings_pipeline = dg.define_asset_job(
name="standings_pipeline_job",
selection=dg.AssetSelection.assets(create_standings_dataframe).downstream(include_self=True),
selection=dg.AssetSelection.assets(create_standings_dataframe).downstream(
include_self=True
),
)

defs_standings: dg.Definitions = dg.Definitions(
Expand All @@ -94,4 +103,4 @@ def defs() -> dg.Definitions:

defs_champions_speed_tiers: dg.Definitions = dg.Definitions(
jobs=[champions_speed_tiers_pipeline],
)
)
66 changes: 44 additions & 22 deletions card_data/pipelines/defs/extract/limitless/extract_standings.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,11 @@
import requests
from bs4 import BeautifulSoup, Tag


def seasons(year: int) -> list[dict]:
url = "https://labs.limitlesstcg.com/"
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
soup = BeautifulSoup(r.content, "html.parser")

season_header = soup.find("h2", string=lambda x: bool(x and f"{year}" in x))
if not isinstance(season_header, Tag):
Expand All @@ -20,11 +21,13 @@ def seasons(year: int) -> list[dict]:
if not isinstance(a, Tag):
continue
parts = list(a.stripped_strings)
tournaments.append({
"name": parts[0],
"date": parts[-1],
"link": f"https://labs.limitlesstcg.com{a['href']}",
})
tournaments.append(
{
"name": parts[0],
"date": parts[-1],
"link": f"https://labs.limitlesstcg.com{a['href']}",
}
)

return tournaments

Expand All @@ -33,22 +36,33 @@ def build_standings(tournament: dict) -> pl.DataFrame | None:
tournament_id = tournament["link"].split("/")[-2]

r = requests.get(tournament["link"])
soup = BeautifulSoup(r.content, 'html.parser')
soup = BeautifulSoup(r.content, "html.parser")

table = soup.find("table", class_="data-table striped")

if isinstance(table, Tag):
headers = ['Rank', 'Name', 'Country', 'Points', 'Record', 'OPW%', 'OOPW%', 'Deck', 'Decklist', 'Unknown']
headers = [
"Rank",
"Name",
"Country",
"Points",
"Record",
"OPW%",
"OOPW%",
"Deck",
"Decklist",
"Unknown",
]

rows = []
tbody = table.find('tbody')
tbody = table.find("tbody")
if not isinstance(tbody, Tag):
return None

for tr in tbody.find_all('tr'):
for tr in tbody.find_all("tr"):
if not isinstance(tr, Tag):
continue
cells = tr.find_all('td')
cells = tr.find_all("td")

if len(cells) == 1:
continue
Expand All @@ -58,29 +72,37 @@ def build_standings(tournament: dict) -> pl.DataFrame | None:
if not isinstance(td, Tag):
continue
if i == 2:
img = td.find('img')
img = td.find("img")
if isinstance(img, Tag):
country = img.get('alt') or img.get('title') or ''
country = img.get("alt") or img.get("title") or ""
row_data.append(country)
else:
row_data.append('')
row_data.append("")

elif i == 7: # Deck column
pokemon_imgs = td.find_all('img', class_='pokemon')
pokemon_imgs = td.find_all("img", class_="pokemon")
if pokemon_imgs:
pokemon_names = [str(img.get('alt', '')) for img in pokemon_imgs if isinstance(img, Tag) and img.get('alt')]
pokemon_string = '/'.join(pokemon_names)
pokemon_names = [
str(img.get("alt", ""))
for img in pokemon_imgs
if isinstance(img, Tag) and img.get("alt")
]
pokemon_string = "/".join(pokemon_names)
row_data.append(pokemon_string)
else:
row_data.append('')
row_data.append("")

elif i == 8: # Decklist column
link = td.find('a')
link = td.find("a")
if isinstance(link, Tag):
decklist_url = link.get('href', '')
row_data.append(f"https://labs.limitlesstcg.com{decklist_url}" if decklist_url else '')
decklist_url = link.get("href", "")
row_data.append(
f"https://labs.limitlesstcg.com{decklist_url}"
if decklist_url
else ""
)
else:
row_data.append('')
row_data.append("")

else:
cell_text = td.get_text(strip=True)
Expand Down
2 changes: 1 addition & 1 deletion card_data/pipelines/defs/extract/tcgcsv/extract_pricing.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
"sm3.5": "2054",
"sm3": "1957",
"sm2": "1919",
"sm1": "1863"
"sm1": "1863",
}


Expand Down
4 changes: 3 additions & 1 deletion card_data/pipelines/defs/extract/tcgdex/extract_series.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ def extract_series_data() -> pl.DataFrame:
raise

filtered = [
s.model_dump(mode="json") for s in validated if s.id in ["me", "sv", "swsh", "sm"]
s.model_dump(mode="json")
for s in validated
if s.id in ["me", "sv", "swsh", "sm"]
]
return pl.DataFrame(filtered)
8 changes: 5 additions & 3 deletions card_data/pipelines/defs/load/tcgcsv/load_pricing.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import subprocess # nosec
import subprocess # nosec
from pathlib import Path

import dagster as dg
Expand Down Expand Up @@ -37,7 +37,7 @@ def data_quality_checks_on_pricing() -> None:
current_file_dir = Path(__file__).parent
print(f"Setting cwd to: {current_file_dir}")

result = subprocess.run( # nosec
result = subprocess.run( # nosec
[
"soda",
"scan",
Expand All @@ -58,4 +58,6 @@ def data_quality_checks_on_pricing() -> None:
print(result.stderr)

if result.returncode != 0:
raise Exception(f"Soda data quality checks failed with return code {result.returncode}")
raise Exception(
f"Soda data quality checks failed with return code {result.returncode}"
)
10 changes: 5 additions & 5 deletions card_data/pipelines/defs/transform/transform_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

DBT_PROJECT_PATH = Path(__file__).joinpath("..", "..", "..", "poke_cli_dbt").resolve()


class CustomDbtTranslator(DagsterDbtTranslator):
def get_asset_key(self, dbt_resource_props):

Expand All @@ -25,18 +26,17 @@ def get_asset_key(self, dbt_resource_props):
# For models, use default behavior
return super().get_asset_key(dbt_resource_props)


@dbt_assets(
manifest=DBT_PROJECT_PATH / "target" / "manifest.json",
dagster_dbt_translator=CustomDbtTranslator()
dagster_dbt_translator=CustomDbtTranslator(),
)
def dbt_build_assets(context: dg.AssetExecutionContext, dbt: DbtCliResource):
"""
dbt assets that transform staging data into final models.
"""
yield from dbt.cli(["build"], context=context).stream()


dbt_resource = DbtCliResource(project_dir=DBT_PROJECT_PATH)
defs = dg.Definitions(
assets=[dbt_build_assets],
resources={"dbt": dbt_resource}
)
defs = dg.Definitions(assets=[dbt_build_assets], resources={"dbt": dbt_resource})
Loading
Loading