All notable changes to this project are recorded here, grouped as
Added, Changed, Fixed, Removed and Security. The project uses
Semantic Versioning. From 1.0.0, releases are
minor and patch only: the API only grows, and a change of results moves
MUNI_RESULTS_VERSION and is named here. Before 1.0.0, any release
could change the API, the ABI and every data format.
muniSegmentIterator_FindBreaksAround gives the boundaries around a
byte offset, the last at or before it and the first after it, without
reading the text from its start: what Backspace, the arrow keys and a
click on a word need. It answers as the iterator would, for grapheme
clusters, words, sentences and lines at every setting, class
overrides and a word segmenter included, and leaves the iterator as
it was. In ordinary text it reads a few code points, so on a long
paragraph of English prose it takes under half a microsecond for
clusters, words and lines, where stepping a fresh iterator from the
start takes milliseconds; sentences back up to the last terminator.
It needs an iterator given its whole text at init
(#278).No results change: the Unicode data is still 18.0.0 and
MUNI_RESULTS_VERSION stays 2, so stored normalized keys, skeletons
and verdicts from 1.1.2 stay valid.
bool now carries
MUNI_NODISCARD, so a compiler warns where the library would drop
one of their answers, and tools/check_nodiscard.py keeps new ones
marked. No call dropped one, so nothing a caller sees changes: same
API, same behavior. The header tests now also check that
MUNI_NODISCARD is not empty in C23
(#276).No results change: the Unicode data is still 18.0.0 and
MUNI_RESULTS_VERSION stays 2, so stored normalized keys, skeletons
and verdicts from 1.1.1 stay valid.
MUNI_NODISCARD was empty under GCC 14 in C23, since GCC 14 reports
its C23 as __STDC_VERSION__ 202000L and the macro waited for
202311L, so a program built with GCC 14 could drop any status the
library returns without a warning. In C the macro now takes
[[nodiscard]] in any dialect after C17 whose compiler has the
attribute. A program that drops a status now gets
-Wunused-result from GCC 14 too, which -Werror makes an error
(#271).No results change: the Unicode data is still 18.0.0 and
MUNI_RESULTS_VERSION stays 2, so stored normalized keys, skeletons
and verdicts from 1.1.0 stay valid.
muniSegmentIterator or a
muniScriptIterator back now checks it first, and refuses one no
init call set up with muni_errorInvalid, leaving it as it was: a
zeroed iterator, one of stray bytes, or one whose counts, flags or
states are out of range. Such an iterator was undefined behavior
before, and a forged script iterator could overflow a buffer on the
stack. muniSegmentIterator_GetLineBreakStrictness and
muniSegmentIterator_GetWritingSystem give their defaults for it, as
for NULL. A copy of an initialized iterator is accepted. The
pointers an iterator keeps (the text, the overrides, the breaker)
cannot be checked, so an iterator changed there by hand stays outside
the contract
(#264,
#267).No results change: the Unicode data is still 18.0.0 and
MUNI_RESULTS_VERSION stays 2, so stored normalized keys, skeletons
and verdicts from 1.0.0 stay valid.
The API is settled. 1.0.0 has the API and the results of 0.13.0; what changes is what a caller may rely on from here. The promise is in What 1.0.0 promises.
harfbuzz.h included, is removed or renamed,
and no release is breaking: a change that would break 1.x is made
additive or does not ship. There will be no 2.0.0
(#257).properties.h are part
of the API, so data may store them; a value a new Unicode version
adds goes after the last
(#257).maul-unicode and maul-unicode-harfbuzz accept
a request for any earlier 1.y, where 0.x accepted only its own
version; the shared library's SOVERSION is 1.0, so a program
built against 1.0 loads any 1.0.z and is built again for 1.1
(#248).No results change: the Unicode data is still 18.0.0 and
MUNI_RESULTS_VERSION stays 2, so stored normalized keys, skeletons
and verdicts from 0.13.0 stay valid.
muniMeasureUtf8 finds the longest prefix of UTF-8 text that ends at
a boundary of a muniTextUnit and holds at most maxBytes bytes and
at most maxUnits units, and gives its length in bytes and in units;
either output may be NULL. It turns a byte offset into a count of
units, turns a count of units into a byte offset, cuts text to fit a
buffer, and counts the grapheme clusters of a whole text or finds
where a count of them ends. A limit that falls inside a code point or
a cluster ends the prefix at the boundary before it, and a count of
UTF-16 units that falls inside a surrogate pair ends it before the
pair. A call that fits whole and asks for no count does not walk the
text
(#243).muni_unitUtf16, a new muniTextUnit, counts UTF-16 units: one for
a code point up to U+FFFF, two for one past it, and one for a maximal
ill-formed subpart, as its U+FFFD. With muniMeasureUtf8 it maps
UTF-8 byte offsets to the UTF-16 offsets that AppKit, UIKit and
Android take, and back
(#243).muni_errorCapacity, muniConvertUtf8ToUtf16,
muniConvertUtf16ToUtf8, muniConvertUtf8ToUtf32 and
muniConvertUtf32ToUtf8 give the input offset of the first code
point not written, instead of the input length. The output holds the
conversion of the input before that offset, always whole code points,
and neededOut still counts the whole conversion. A caller can keep
what was written, or convert the rest from the offset, instead of
converting again with less input until it fits. A strict-mode error
after the output filled up still returns that error and its offset.
Code that read the input length from the offset on
muni_errorCapacity uses its own length
(#241).muniCutUtf8 is removed. muniMeasureUtf8 with
maxUnits set to SIZE_MAX gives the same cut; the unit now comes
before the byte limit. No consumer called it
(#243).| 0.12.0 | 0.13.0 |
|---|---|
muniCutUtf8(text, length, budget, unit, &cut) |
muniMeasureUtf8(text, length, unit, budget, SIZE_MAX, &cut, NULL) |
muniSegmentIterator_GetWordKind tells what the segment ending at the
boundary a word iterator last reported holds, as a muniWordKind: no
word (spaces, punctuation, symbols), a number, letters, kana or
ideographs, the highest kind among its code points. Word movement and
double-click selection skip segments of kind muni_wordNone. Letters
with digits are letters, and a superscript digit such as "²" makes no
number. ICU's word rule statuses agree on segments of letters or
digits alone; on a mixed segment ICU reports the last rule it matched,
so it calls "ab12" a number
(#231).muniGetDecomposition writes the full decomposition of one code
point, canonical (muni_nfd) or compatibility (muni_nfkd), every
mapping applied, in canonical order; Hangul syllables give their jamo
and a code point that does not decompose gives itself. It reports the
count it needs, with muni_errorCapacity when the output is too
small. MUNI_MAX_DECOMPOSITION (18, for U+FDFA in NFKD), now public
in normalize.h, always suffices for this Unicode version; a release
with a new Unicode version may raise it
(#227).muniDecodeUtf16, the counterpart of muniDecodeUtf8, decodes the
code point at the start of UTF-16 units: a surrogate pair takes two
units, anything else one. An unpaired surrogate gives
muni_errorUtf16Surrogate with U+FFFD and one unit, as
muniConvertUtf16ToUtf8 replaces it, so a decoding loop always makes
progress
(#228).muniCutUtf8 gives the length of the longest prefix of UTF-8 text
within a byte budget that ends at a boundary of a unit:
muni_unitCodePoint, where each maximal ill-formed subpart counts as
one, or muni_unitGraphemeCluster, which keeps a letter with its
accents, an emoji ZWJ sequence or a flag whole, for text fields and
preedit strings. The unit is an enumeration so that a later release
can add words
(#230).muniSegmentIterator_EnableCjAsIdeographic(iterator,
true) becomes muniSegmentIterator_SetLineBreakStrictness(iterator,
muni_lineBreakLoose), which also adds loose's other breaks.
muniSegmentIterator_IsCjAsIdeographicEnabled is removed;
muniSegmentIterator_GetLineBreakStrictness reports the level. A new
iterator breaks as before
(#221).line-break property:
strict, normal (the default), loose and anywhere, set with
muniSegmentIterator_SetLineBreakStrictness before the first break.
muniSegmentIterator_SetWritingSystem with
muni_writingChineseJapanese turns on the rules CSS keeps for Chinese
and Japanese. Loose breaks before small kana and the prolonged sound
mark (Line_Break=CJ), before iteration marks and between characters
such as two ellipses; anywhere breaks between any two grapheme
clusters. Normal and strict keep CJ with what precedes it, as the CSS
Text 3 Candidate Recommendation Draft of 14 August 2026 says; ICU4C's
normal rules and earlier CSS drafts break before CJ in normal. The
default level breaks as before, at nearly the same cost (0.15% more
instructions on the bench)
(#221).muniConvertCase takes two more languages. muni_caseGreek
uppercases Greek as it is written in capitals: accents and breathings
go, a dialytika stays and appears where a dropped accent kept two
vowels apart ("Μάιος" to "ΜΑΪΟΣ"), the disjunctive eta "ή" keeps its
tonos when it stands alone, and an iota subscript becomes a capital
iota; it follows ICU's Greek rules. muni_caseArmenian uppercases
the ligature "և" to "ԵՎ" and titlecases it to "Եվ". Lowercasing,
titlecasing (but the ligature) and folding follow the default rules
(#223).muniConvertCase with
muni_caseGreek or muni_caseArmenian, which were refused, now give
results. No result of the earlier languages changes, so stored keys
and skeletons stay valid; a caller that stores the results version
sees it move once
(#223).muniReorderBidiLine leaves runsOut and countOut as they were on
muni_errorInvalid, and muniGetResolvedScripts leaves scriptsOut
and countOut as they were on muni_errorInvalid and on a UTF-8
error. Both calls always behaved this way
(#204).base.h now says what the results version tracks: the status and
offset of the calls it covers, and their output on success, but not
the bytes and counts a refusal leaves. This is why the change to
refused skeletons below keeps it at 1
(#206).muniGetBidiClass returns Left_To_Right for every value past
U+10FFFF, as properties.h says. It returned Boundary_Neutral, the
class of the noncharacters U+10FFFE and U+10FFFF, which the table
carried past them. Only a caller that passes a value that is no code
point sees the change: bidi and skeletons read the class only of
code points decoded from valid text, so their results and the results
version are unchanged
(#209).muniGetSkeleton now stops at the segment that hit the limit when it
refuses text with muni_errorLimit (a run of combining marks past
MUNI_MAX_NORMALIZATION_SEGMENT). It writes the skeleton of the text
shown before that segment, as far as textOut holds it, and
neededOut gives its length in bytes; nothing from that segment on is
written. Before, it went on past the run, dropped the marks that did
not fit, wrote the skeleton of the rest of the text and set
neededOut to 0. muniNormalize and muniGetNfkcCasefold already
stopped this way. A skeleton that succeeds is unchanged, and so is
the results version
(#203).int32_t instead of uint8_t, as the Maul family's rule for
enumerations asks, and muniBreakClassOverride.value is an
int32_t. Their values and numbers are unchanged. This changes the
binary interface, so a program built against 0.8.0 is built again;
code that stores these values in a uint8_t keeps working, since
every value fits. A negative value, such as -1, is refused with
muni_errorInvalid, as a value past the last one is. The 16 types
are muniBidiDirection, muniConvertMode, muniNormalForm,
muniQuickCheck, the nine property kinds (muniGeneralCategory to
muniBracketType), muniRestrictionLevel, muniCaseOperation and
muniCaseLanguage. muniScript stays a uint32_t, since it is a
packed script tag, and bidi levels stay uint8_t
(#196).| 0.8.0 | 0.9.0 |
|---|---|
typedef uint8_t muniBidiDirection (and the 15 other kinds) |
typedef int32_t muniBidiDirection |
muniBreakClassOverride.value: uint8_t |
int32_t |
MUNI_SCRIPT is renamed MUNI_MAKE_SCRIPT, as the
family names a helper that makes a value:
MUNI_SCRIPT('L', 'a', 't', 'n') becomes
MUNI_MAKE_SCRIPT('L', 'a', 't', 'n'). It is still a macro, so it
works in constant expressions. MUNI_SCRIPT_COMMON,
MUNI_SCRIPT_INHERITED and MUNI_SCRIPT_UNKNOWN keep their names
(#196).muniNormalize, muniConvertCase, muniGetNfkcCasefold,
muniGetSkeleton and muniInvertBidiMap refuse an output that
overlaps their input with muni_errorInvalid, before they write
anything. Before, transforming text in place returned
muni_success with a corrupted result, and inverting a map in place
destroyed the map. Text cannot be transformed in place: write the
result to another buffer. A call that kept its input and output
apart behaves as before, and results are unchanged
(#191).muniInvertBidiMap clears inverseOut over its whole count when it
refuses a map with an entry out of range or a repeated entry. It used
to leave a mix of its working marks and partial entries, which the
header called unspecified. A NULL argument or an inverseOut that
overlaps the map still leaves it as it was
(#192).muniSegmentIterator_NextLineBreak said a
muni_errorBusy refusal sets mandatoryOut to false. The call
leaves it as it was, as it always did, and the header now says so
(#179).include/maul-unicode/base.h now
configures an existing build directory again, so the CMake project
version, the package version file and the pkg-config file follow it.
Before, they kept the version of the first configure
(#200).muniGet..., as the Maul
family's naming rule asks of functions that act on no object. Their
parameters, results and the results version are unchanged: code that
calls them changes the name and nothing else. The engine and
maul-locale each rename these calls once, with their next pin of
maul-unicode
(#161).| 0.7.0 | 0.8.0 |
|---|---|
muniToLower |
muniGetLowercase |
muniToUpper |
muniGetUppercase |
muniToTitle |
muniGetTitlecase |
muniToNfkcCasefold |
muniGetNfkcCasefold |
muniResultName is now
muniGetResultName, as the Maul family's naming rule asks. Their
parameters and behavior are unchanged: code that calls them changes
the name and nothing else. maul-locale moves to the new names with
its next pin of maul-unicode
(#150).| 0.6.0 | 0.7.0 |
|---|---|
muniResultName |
muniGetResultName |
muniFeedSegmentIterator |
muniSegmentIterator_Feed |
muniNextSegmentBreak |
muniSegmentIterator_NextBreak |
muniNextLineBreak |
muniSegmentIterator_NextLineBreak |
muniSetComplexBreaker |
muniSegmentIterator_SetComplexBreaker |
muniSetBreakClassOverrides |
muniSegmentIterator_SetBreakClassOverrides |
muniSetLineOptions(iterator, muni_lineCjAsIdeographic) |
muniSegmentIterator_EnableCjAsIdeographic(iterator, true) |
muniFeedScriptIterator |
muniScriptIterator_Feed |
muniNextScriptRun |
muniScriptIterator_NextRun |
The initializers (muniInitLineIterator and the others), the
muniFind* calls, muniEncodeUtf8 and every other text and code
point function keep their names.
- Breaking: whether a line iterator breaks before Line_Break=CJ
characters, such as the small kana and the prolonged sound mark, is
now a switch, which muniSegmentIterator_EnableCjAsIdeographic turns
on with true and off with false. It is off in a new line iterator, which
then follows the default rules of UAX #14, as an iterator without
options did before. As the option was, it is set before the first
break and refused for any iterator that is not a line iterator
(#150).
muniSegmentIterator_IsCjAsIdeographicEnabled tells whether a line
iterator resolves CJ as ID. It gives false for a NULL iterator and
for one that is not a line iterator
(#150).muniSetLineOptions and muni_lineCjAsIdeographic.
Use the switch above: muniSegmentIterator_EnableCjAsIdeographic
with true in place of the option, and with false in place of 0
(#150).muniComplexBreakFn
is renamed muniComplexBreakFunction, the name the Maul family gives
its callback types. Its parameters and contract and
muniSetComplexBreaker are unchanged: code that names the type
changes the name, and code that only passes a function changes
nothing
(#130).muniEncodeUtf8 takes the output's capacity after
bytes: muniEncodeUtf8(codePoint, bytes, capacity, sizeOut).
Before, it wrote up to four bytes without knowing the array's size,
so a caller encoding into the rest of its own buffer that checked the
room wrongly had bytes written past its end. Now a capacity below the
encoding's length returns muni_errorCapacity, writes no byte and
puts the length needed, 1 to 4, in sizeOut; a NULL bytes with
capacity 0 measures. A surrogate, a value past U+10FFFF, a NULL
sizeOut, or a NULL bytes with a nonzero capacity returns
muni_errorInvalid and leaves both outputs as they were
(#129).MUNI_MAX_NORMALIZATION_SEGMENT (32) in normalize.h names the
bound behind every muni_errorLimit: a normalization segment, a
starter and the marks after it, has at most 32 code points, so 31
marks may follow a starter. muniNormalize, muniToNfkcCasefold and
muniGetSkeleton refuse a longer one, as before
(#108).MUNI_MAX_SCRIPT_EXTENSIONS (32) in properties.h and
MUNI_MAX_RESOLVED_SCRIPTS (64) in security.h: an array of that
size always holds what muniGetScriptExtensions and
muniGetResolvedScripts write for this release's Unicode version. A
later Unicode version may raise them; the calls still report the
count they need and return muni_errorCapacity instead of writing
past a smaller array
(#119).MUNI_RESULTS_VERSION and muniGetResultsVersion. The version moves
up with every change to what muniNormalize, muniToNfkcCasefold,
muniConvertCase or muniGetSkeleton give, or to the verdict of
muniCheckIdentifier, muniGetRestrictionLevel,
muniCheckMixedNumbers or muniGetResolvedScripts, a new Unicode
version included. Store it beside normalized keys and skeletons, and
recompute them when it changes: a fixed rule within one Unicode
version changes results without changing muniGetUnicodeVersion.
It starts at 1, and no result changed in this release
(#121).muni_errorInvalid (-1) and muni_errorCapacity (-2) keep their
values, muni_errorLimit moves from -10 to -9, and the library's own
errors move to -32 to -40: the six UTF-8 errors at -32 to -37 in
their old order, then muni_errorUtf16Surrogate -38,
muni_errorUtf32Value -39 and muni_errorIdentifier -40. Code that
uses the names needs only a rebuild; code or data that stores the
numbers must change them
(#108).muni_errorState (-6) instead of muni_errorInvalid:
muniSetComplexBreaker, muniSetBreakClassOverrides and
muniSetLineOptions on an iterator already asked for a break or
boundary, and muniFeedSegmentIterator and muniFeedScriptIterator
on an iterator that did not ask for more text. A wrong argument stays
muni_errorInvalid in any state, and is reported first
(#111).muniDecomposePair returns a muniResult and takes a
countOut after secondOut, which receives 0 when the code point
does not decompose, 1 for a singleton and 2 for a pair. A NULL output
returns muni_errorInvalid and writes nothing. Before, it returned
false both for a code point without a decomposition and for a NULL
output, and a singleton was told from a pair only by a second code
point of 0
(#117).muniSetComplexBreaker that
calls back into its own iterator gets the new muni_errorBusy (-10)
from the three setters, muniFeedSegmentIterator,
muniNextSegmentBreak and muniNextLineBreak, and the call changes
nothing. Before, such a call worked on the iterator's state from
before the outer muniNextSegmentBreak or muniNextLineBreak, which
then overwrote it, so a change was lost without notice or a break
came from stale state. The segmenter must not initialize its own
iterator
(#115).muniGetSkeleton takes the workspace's size as a new
argument after workspace. A workspace smaller than 2 * length
returns muni_errorCapacity with offset 0 and neededOut 0, before
the text is read, for all text: before, a short workspace was written
past its end only when the text needed bidi, so the mistake passed
tests with Latin text. A too-small output always has neededOut
above its capacity, so the two errors can be told apart. workspace
may be NULL only when workspaceSize is 0; for text that is not
empty, that returns muni_errorCapacity instead of
muni_errorInvalid
(#92).muniReorderBidiLevels and muniReorderBidiLine return
muni_errorInvalid for a level above 126, the most rules I1 and I2
leave, and muniReorderBidiLine for a paragraph level above 1, before
writing anything. Before, a level such as 200 was ordered and
returned muni_success
(#100).size_t, which a 32-bit target can
reach through UTF-16 to UTF-8 conversion, case mapping,
normalization, NFKC_Casefold or skeletons, returns
muni_errorCapacity with neededOut set to SIZE_MAX. Before, the
count wrapped and the call reported a size too small or returned
muni_success with only part of the result written
(#87).muniInvertBidiMap returns muni_errorInvalid for a map that
repeats an entry, such as {2, 0, 2}. Before, it returned
muni_success with one slot of inverseOut never written
(#89).muniReorderBidiLine writes no runs when they do not fit, as
documented. Before, it returned muni_errorCapacity with the runs
that fit left in the output in logical order
(#99).muniGetSkeleton sets neededOut to 0 on every error except an
output too small, which reports the size the skeleton needs, and
muni_errorInvalid, which leaves it as it was. Before,
muni_errorLimit left the bytes counted before the limit, which
tells a caller nothing since the skeleton follows display order
(#101).muniCheckNormalization validates the whole text and reports a UTF-8
error after the point where the answer became No. Before, it stopped
there and returned muni_success
(#106).muniResolveBidi, and muniGetSkeleton for text that needs
bidi) resolves an FSI's direction only when the isolate takes a
level, as UAX #9 rules X5a to X5c have it: an isolate past the depth
limit of 125 takes no level, so its direction is never used. Every
result is unchanged
(#79).muniIsIdeographic tells whether a code point has Unicode's
Ideographic property (PropList.txt), which a word rule over ideographs
such as CLDR's Japanese one needs. It is not Script=Han: Han includes
the CJK radicals, which are not Ideographic, and leaves out Tangut,
Khitan Small Script, Jurchen, Nushu and Small Seal, which are. A value
past U+10FFFF is false. No existing result changes
(#63).-msse2 -mfpmath=sse,
from the family's compiler flags), so it needs a processor with SSE2;
results are the same
(#57).muniScriptIterator is 208 bytes on every target, from
224: its storage now fits the state it holds on 64-bit targets,
where the state is largest, instead of keeping 16 spare bytes there.
Programs must be rebuilt against the new headers; no
source change is needed
(#54).muniSetBreakClassOverrides refuses a count over 0x10F800, one entry
per scalar value, before it reads any entry, and segment.h
documents that limit. The old check against UINT32_MAX never fired
where size_t is 32 bits. GCC 14 warned about it there
(-Wtype-limits), which stopped builds that treat warnings as errors
(MAUL_UNICODE_WERROR). A larger count is still muni_errorInvalid,
and valid tables behave the same
(#50).Word, sentence and line segmentation can follow a language's conventions where locale data such as CLDR gives a character another class.
muniSetBreakClassOverrides), for conventions such as Greek, which
ends a sentence at U+003B and U+037E, and POSIX English, which keeps
the full stop out of words but not out of numbers. The table is the
caller's, sorted by code point and read in place.muniSetLineOptions).
muni_lineCjAsIdeographic allows a line break before the small
kana, the prolonged sound mark and the other Line_Break=CJ
characters, as Chinese and Japanese line breaking does.libmaul-unicode.so.0.1.0 instead of .so.0.0, so a program is
never loaded with another 0.x release than the one it was built
against.The first release: Unicode algorithms and character properties from Unicode 18.0.0, passing every official conformance file for the algorithms it implements.
maul-unicode-harfbuzz, which gives
HarfBuzz its Unicode data from the library.clang-cl, and WebAssembly.