Problems

A problem is a task solved by submitting a program, which is then run against a set of tests. Several parts make that up: the text people read, the data their program runs on, the limits it runs under, and the program that decides whether its output is right. Eolymp calls the collection of problems in a space the catalog, and the API behind it is atlas.

The problem itself holds little more than metadata. Everything substantial hangs off it — statements, testsets and tests, generators, a checker, optionally an interactor and a validator, reference solutions, code templates, attachments, an editorial — and each of those has its own service. This page describes the pieces and the API for each.

Overview

Problems belong to a space, so they are created against the space URL:

https://api.eolymp.com/spaces/<space-id>/problems

Everything below that belongs to a single problem and is addressed relative to it, at https://api.eolymp.com/spaces/<space-id>/problems/<problem-id>. A problem carries that address in its read-only url field, and the SDK idiom is to take it from there rather than build strings by hand:

problems = eolymp.atlas.ProblemServiceClient(transport, url=space_url)
problem = problems.DescribeProblem(request=eolymp.atlas.DescribeProblemInput(problem_id=problem_id)).problem

# every other service is constructed against the problem, not the space
testing = eolymp.atlas.TestingServiceClient(transport, url=problem.url)
statements = eolymp.atlas.StatementServiceClient(transport, url=problem.url)

Reads need the atlas:problem:read scope and writes need atlas:problem:write. A few reads — problem history, checker and validator configuration, anything that can expose author secrets — require the write scope instead.

The problem

CreateProblem adds a problem and returns its id. A first statement can be passed along with it, or statements can be added later.

out = problems.CreateProblem(request=eolymp.atlas.CreateProblemInput(
    problem=eolymp.atlas.Problem(
        type=eolymp.atlas.Problem.PROGRAM,
        number=101,          # position in the catalog
        difficulty=2,        # 0 (very easy) to 5 (very hard)
        visible=False,       # keep it out of the catalog until it's ready
    ),
    statement=eolymp.atlas.Statement(
        locale="en",
        title="Sum of Two Numbers",
        content=eolymp.ecm.Content(markdown="Read two integers and print their sum."),
    ),
))

type decides what solving means: PROGRAM for the usual write-a-program task, FUNCTION when the participant supplies a function rather than a whole program, OUTPUT for output-only problems where the participant uploads an answer file, INTERACTIVE when the participant's program talks to an interactor instead of reading a static input file, plus SQL and ML.

UpdateProblem changes metadata, and it is also how a problem gets published — by flipping visible. It takes a patch mask, and an empty mask writes every field, blanking anything left empty, so name the fields intended. Content lives in other services and is never touched here. DeleteProblem removes the problem and everything attached to it with no undo; to retire a problem while keeping it, make it invisible instead.

Setting origin on creation does something different: instead of an empty problem the result is a copy of one from Polygon, from Basecamp, or from another space. The copy happens in the background, so content appears some time after the call returns, and SyncProblem later pulls the origin again — overwriting local edits.

Statements

A statement is the problem text for one locale. A problem normally has one per language, and publishing in another language means creating another statement rather than editing the existing one. Content is Markdown or LaTeX.

Sections come from markers on lines of their own — \InputFile, \Interaction, \OutputFile, \Examples, \Note, \Scoring — and whatever precedes the first marker is the introduction.

CreateStatement adds a locale, UpdateStatement patches one, and LookupStatement fetches by locale with fallback to another when the requested one does not exist — the method to use when rendering for a reader. TranslateStatements starts machine translation into other locales in the background; the results are marked as automatic, and hand-written statements are not overwritten unless requested.

Statement content is returned only on request, either as raw markup for editing or as a parsed tree for display. Pictures go through the Asset API — upload, then use the URL in the Markdown or LaTeX.

Testing configuration

Two layers of settings decide how a solution is run.

The problem-wide TestingConfig holds the defaults: time_limit in milliseconds, cpu_limit, memory_limit in bytes, the problem type, and run_count if a solution should be run more than once per test. It has no patch mask — read it with DescribeTestingConfig, change what is needed, send the whole thing back.

Testsets carry the limits that actually apply to their runs. Read the effective limits from the testset, not from the problem config.

Testsets

A testset is a group of tests evaluated together — a subtask, in olympiad terms. CreateTestset makes an empty one, and its index both orders it within the problem and is how other testsets refer to it.

Each testset has its own time_limit, cpu_limit, memory_limit and file_size_limit, plus two further settings described below.

scoring_mode decides how the testset's points are worked out:

feedback_policy decides how much of the result the participant sees: COMPLETE shows every test, ICPC shows only the first test that was not accepted, and ICPC_EXPANDED shows that plus its number.

Testsets can also depend on others through dependencies, listing them by index. A testset with dependencies stays unevaluated until they pass, where "pass" means either all of them fully accepted or any of them scoring at least one point, depending on dependency_mode.

Tests

A test is an input and the expected answer. CreateTest adds one to a testset, and each half can arrive in one of three ways:

• inline, as input_content / answer_content
• as a URL, input_url / answer_url — upload the file through the Asset API and pass the link
• as a generator invocation, input_generator / answer_generator

Generated data is produced in the background, so a test stays PENDING until generation and validation succeed, then becomes READY, or INVALID if something went wrong — status_message says what. A submission that arrives before generation has finished forces generation to happen inline as part of that evaluation.

The rest of a test is flags and numbers: score for the points it is worth, index for its order, example to show it in the statement's \Examples section, secret to ensure its data is never returned by the API, and inactive to keep a test without running it.

Interactive problems differ here: the stored input and answer are instructions for the interactor rather than data a reader would recognise. The example overrides address that — example_input_content and example_answer_content change what the statement displays without touching what gets run.

ListExamples returns the example tests across all testsets and needs only read access, which makes it the one method here that can be called from a participant-facing view.

Generators

Generators are what the API calls scripts. A generator is a program that prints test data to standard output; a test invokes it by name with arguments, such as gen 200000 42.

An answer generator additionally receives the input data on its standard input, so the author's own solution works as an answer generator with no changes.

CreateScript stores one, with a name, a runtime and its source. Tests reference generators by name rather than by id, so renaming or deleting one means fixing up the tests that still invoke it — a command naming a generator that no longer exists has no way to produce data.

ExecuteStressCheck runs a generator over and over and puts the data it produces through the problem's solutions, which is how to hunt for the case that breaks something. It runs in the background and reports through the problem's activity feed.

Checkers

The checker decides whether output is correct. It is configured with UpdateChecker, which replaces the configuration in full, and its type picks the strategy:

A program checker also gets EOLYMP=1, INPUT_FILE, OUTPUT_FILE and ANSWER_FILE in its environment, and can carry extra files placed in its working directory. It runs only for solutions that finished cleanly — a program that crashed, ran out of time or overran memory has already failed.

There is no DeleteChecker. To go back to plain comparison, update the checker to LINES. A problem with neither a checker nor an interactor cannot grade anything.

Setting secret on a checker keeps its source hidden from anyone who may not see problem secrets. DescribeChecker returns source code, which is the reason to set it.

Interactors

An interactive problem replaces the read-input-print-output contract with a conversation. The interactor starts on every test just before the solution, with the two processes' stdin and stdout wired to each other, and is configured through UpdateInteractor.

Its exit code decides the test: 0 means the interaction went fine and the checker grades the result, 1 fails that test as a wrong answer, and anything else fails the whole submission as a system error.

There is no DeleteInteractor either — call UpdateInteractor with an empty payload to remove it. Configuring one for the first time needs the problem's type set to INTERACTIVE already; UpdateInteractor refuses otherwise, naming the fix. Removing an interactor has no such restriction, but changing type away from INTERACTIVE does — that is refused for as long as an interactor is still configured, so the interactor has to go first.

Trying a solution without submitting it — CreateRun, behind the "Run" action on the archive, contest and course editors — can target one of the problem's own example tests by id (example_id) instead of typed-in input, and that works for a problem of any type. It matters most for INTERACTIVE problems, where a run also attaches the interactor: the submitted program and the interactor talk to each other exactly as they would during a real submission, and the output comes back unchanged, with no separate rendering for the interactor's side of the exchange. A non-interactive problem never gets an interactor attached, even for an example-based run, and an INTERACTIVE problem with none configured yet fails the run outright rather than running without one.

Validators

A validator checks that test inputs are what the statement promises: the right number of numbers, within the stated bounds, in the stated format. It never looks at the answer. It is the piece authors most often skip and most often regret skipping, because a malformed input usually shows up as a mysteriously wrong verdict much later.

It has two halves:

• UpdateValidator stores the validator with the problem. Saving it does not run it. As with the others, an empty payload removes it.
• RunValidation runs a validator across every test of the problem. It reads the validator straight out of the request rather than from storage, so a candidate can be tried before being committed.

Validation is asynchronous: RunValidation returns an id, and DescribeValidation reports progress test by test until it settles. The per-test verdict distinguishes an input the validator rejected from one that could not be produced at all because a generator or download failed. Starting a validation cancels any earlier one still running.

Reference solutions

A solution here is the author's own program stored together with the verdict it is expected to get — accepted, wrong answer, timeout, and so on. Solutions demonstrate that the problem grades as intended: a solution meant to time out should time out, and if it does not, the limits are wrong.

CreateSolution stores one without running it. CheckSolutions submits them against the problem's tests and records whether each got the verdict it was supposed to get. It returns immediately and reports nothing itself, so poll ListSolutions until nothing is pending.

Templates, attachments and editorials

Three smaller services complete a problem.

CodeTemplateService holds the starting code a participant sees when first opening the editor — usually the boilerplate for reading input and printing output — at most one template per runtime. A template can also wrap hidden code around the participant's code before compilation and place extra files in the working directory. Grading logic must never live in a template: in several languages a running program can read those files, so grading logic belongs in the checker or interactor.

AttachmentService registers the extra files published alongside a problem. Upload through the Asset API first, then register the link here.

EditorialService holds the write-up explaining the solution, stored per locale like a statement and using the same Markdown-or-LaTeX content model.

Putting a problem together

The pieces are independent, but they do have a natural order:

1. Create the problem, with or without its first statement.
2. Add statements for the required locales.
3. Set the problem-wide testing configuration, then create testsets with their limits, scoring and feedback.
4. Add generators, if the data is produced rather than uploaded.
5. Add tests, flagging the ones that should appear in the statement as examples.
6. Configure the checker, and an interactor if the problem is interactive.
7. Add a validator and run it over the tests.
8. Add reference solutions and run CheckSolutions.
9. Make the problem visible.

Steps 4, 5, 7 and 8 all involve background work, so poll rather than assume that a call which returned has finished its job.