Implement storage limits for local repository

Currently, the local repository doesn't implement any storage limits and it's possible to fill up the entirety of the users drive.
We need to add:

[x] Some sort of storage quota mechanism

[ ] Percentage based - user designates a % of bytes available to codex
[x] Exact amount - user designates the exact number of bytes available to codex

[ ] Pruning of old chunks from the system when some threshold has been reached

[ ] This requires running a periodic task that cleans up old chunks

[ ] More??

while working on the BlockStore API, I found that current code doesn't use delBlock at all, meaning that local store can only grow :)  So I think it's also important feature for any practical usage, no less that SQLite backend.

Yes, this is definitely an important feature for a usable product, so this should be relatively high on our priority list.

Yes, all issues that currently in our priority list are assigned to me or @michaelsbradleyjr, so we will take care of it.
If this simple approach will go out of control, we can create milestone for them.

I have started researching some possibilities.
For SQLiteStore introduced in #160 we could make use of SQLite's DBSTAT Virtual Table.
The SQLITE_ENABLE_DBSTAT_VTAB compile-time option seems to already be enabled in our builds via nim-sqlite3-abi, e.g. this query works for a database created by an instance of type SQLiteStore
SELECT pgsize FROM dbstat WHERE name='Store' AND aggregate=TRUE;

The result is "the total amount of disk space used" in bytes. Note that "total" doesn't include "Freelist pages, pointer-map pages, [or] the lock page", so numbers reported by e.g. ls may include some overhead.
We could periodically query the dbstat info and if the database is over quota, we could sum the data size for the oldest N rows (100 or 1000 or whatever) using the timestamp column that's already part of the Store table created by nim-datastore. Repeat M times until we've reached the amount of space we want to clear within some delta, then delete the oldest N * M rows.
That would require some custom queries, but nim-datastore exports the necessary fields and helpers so that it's entirely doable.
We would need to do some experimentation in relation to VACUUM and whether we want/need to set auto_vacuum=FULL.
I don't have an idea yet for the best approach for FSSstore, but if we end up settling on SQLiteStore then maybe we won't need to worry about it.
I'm not aware of any general cross-platform solution for a percentage-based quota. For the platforms we primarily target (Linux, macOS, Windows) we can probably shell out and use cli utilities in the PATH, and log a warning if the expected utility/ies were not present, errored, or otherwise gave a result that Codex is unable to parse.

Doesn't SQLite provide some native facility to limit the DB size?

Doesn't SQLite provide some native facility to limit the DB size?

even if it does, we need solution deleting LRU blocks rather than hard limit that will just prevent adding new data to DB.

Taking another crack at this from an API perspective.
This is a sketch, not getting into details of how to deal with cli params, whether the API should be method or proc, or other specifics.
What if for type BlockStore we have
method quota(self: BlockStore, percentage: Percentage): Future[?!int]

method quota(self: BlockStore, bytes: int): Future[?!int]

Where the return value is the number of bytes over/under quota.
In the case of percentage: Percentage, the param value indicates the max percentage of storage space that the blockstore should be able to consume relative to the current total size of the filesystem on which it resides.
In the case of bytes: int, the param value indicates the max size of the blockstore.
How the return value is calculated would depend on the implementation of ref object of BlockStore and possibly the target OS.
And we could also have
method purge(self: BlockStore, bytes: Positive, delta: Percentage): Future[?!Natural]

Where the return value is the number of bytes actually purged.
The bytes: Positive param value indicates how many bytes should be purged.
The delta: Percentage param value indicates a percentage range +/- for the return value that is considered acceptable/expected relative to bytes: Positive. If the value to be  returned is outside the range, a warning could be logged by method purge prior to return.
How the purge operation is performed would depend on the implementation of ref object of BlockStore and possibly the target OS.
It would be expected that purge implementations would take into consideration some LRU metadata for blocks, e.g. derived from extra data in the database or the filesystem.
For SQLiteStore that would likely involve using a Datastore implementation (maybe Codex-specific) that derives from type SQLiteDatastore and has an extra column in the Store table. That's certainly within the realm of possibility, and I have some idea for that, but the specifics can be explored later or in another context.

Something to keep in mind: there are also blocks that we should keep around because they're part of a contract that we're engaged in. Not sure whether these should go into a separate store, or that the store itself should treat them differently.

Yeah, contracts and these restrictions contradict, so we may choose between:

ignore contracts and just delete extra blocks
ignore restrictions and just keep extra blocks
make contracts know about the restrictions

Also, we essentially have two parts in blockstore, and their sizes probably should be regulated independently:

permanent (contracted) part that shouldn't be deleted in any circumstances
variable (caching) part

The Sales module only sells storage that it's been told that it is available. So we only get into contracts for disk space that we explicitly marked as available.
So you can let contracts know about the restrictions.

But for the starter, let's develop the system ignoring contracts problem. I think we can use any of three approaches:

active cleaner, working as permanent thread in b/g, cleaning data from time to time
passive cleaner, starting to delete blocks once we have too much
combined approach - once putBlock found we have too much blocks, start b/g GC thread

As far as we have "last access time" for each block, we can run DELETE queries requesting to delete N blocks with oldest timestamps. I think this solves simplfied (contract-ignoring) problem for SQLite backend. So, the algorithm is:

once putBlock found that we have too much blocks
run b/g thread requesting to DELETE x blocks with oldest time
both getBlock and putBlock should unconditionally update "last access time"
we hope that SQLite will reuse space in DB file, otherwise user will need to run DB compaction from time to time

So you can let contracts know about the restrictions.

@markspanbroek  I think about adding to BlockStore a request availableContractSpace: int64 or isAvailableContractSpace(int64), so when you get opportunity to fill a slot, you consult with BlockStore to check whether we can participate. What you think about it?
Sidenote: we can have such checks with other subsystems too (CPU, Network and disk bandwidth required for PoRs).

I would rather have a way to reserve space in the BlockStore. I expect a user would want to dedicate an amount of storage for sale explicitly. There's a REST API endpoint for doing that in the codex node already. When the user indicates that it wants to sell e.g. 100 Gb, the store should always keep this space available for contracts to use.

For now we simply need something that will refuse to write past a specific limit, without doing any automatic deletion or cleanup.
What I would like to see as a first step is:

Allow reporting the total amount of bytes used

This will be used by the sales module to report how much we still have available for sale

Refuse to write passed an upper limit

This will prevent Codex from using up all the available space on a users hard drive

@dryajov this approach will stop caching of new data once we reached the limit. So, yes - it will work, but data dpownloaded from network will be not cached anymore. In particular, #171 (prefetching) will not work.

@dryajov this approach will stop caching of new data once we reached the limit. So, yes - it will work, but data downloaded from the network will be not cached anymore. In particular, #171 (prefetching) will not work.

Thats correct, I want the minimum possible set of functionality right now, we can evolve this iteratively. In particular, it should be fine to stop storing new blocks once we reach the limit.

In particular this means that ECC will became useless - we rely on storing decoded data in the local store.

In particular this means that ECC will became useless - we rely on storing decoded data in the local store.

How so? We're only talking about limiting storage use, nothing else. Sure some files would not be downloadable in full due to size, but this is already true - you can't download more than the size of the hard drive.
I would like to avoid all the complexity associated with "hot" pruning (client is running) for now and focus on a simple and concrete task. We'll come back to this once we have a better understanding of how the software behaves and what sort of real requirements we are working with.
As a second step, we can do "cold" pruning (client is stopped), which should give us a good understanding of what sort of issues we're going to be dealing with once we move to "hot" pruning.

How so?

Recall this discussion about erasureJob:

We spawn erasureJob or prefetchBlocks and then go to download data sequentially by block CIDs via StoreStream
So, if the all original data are fine, we will not prefetch data, and just receive them in slow sequential way via StoreStream
but if some original block is lost, we can't restore it because we rely on sending recovered blocks via local store

How so?

Recall this discussion about erasureJob:

We spawn erasureJob or prefetchBlocks and then go to download data sequentially by block CIDs via StoreStream
So, if the all original data are fine, we will not prefetch data, and just receive them in slow sequential way via StoreStream
but if some original block is lost, we can't restore it because we rely on sending recovered blocks via local store

So, once we stopped saving data from network to local store, we may get very slow downloads and ECC will definitely stop working. Prefetcher/decoder overall rely on our ability to store entire file in cache.

Yes, thats expected, I never meant this issue to handle more complex cases than simple limits on the total amount of bytes the repo can occupy. We can live with some level of degraded functionality once we have reached our storage quota, the alternative being simply stopping the process.
Doing proper hot pruning is not an easy task which overall buys us very little at this point. We should improve it, but I don't see any pressing need to do it immediately.

From discussion with Dmitry:

Implement a hard limit that stops writing new data (putBlock) once the limit is reached
Compute current disk usage by b/g task executed ~~ once per second (just sum up size of SQLite files)
Return to the marketing team the amount of disk quota not yet used - add API node.unusedDiskQuota()

@emizzle will it work for you?

Return to the marketing team the amount of disk quota not yet used - add API node.unusedDiskQuota()

I'm not sure if this node > sales info would be useful. Instead, perhaps the information should flow in the opposite direction, such that the sales modules can inform the datastore how much space it is using in established contracts or wants to use in future contracts, ie "reserve" the space in the datastore?
As @markspanbroek pointed out, currently a host specifies how much space they want to make available to sell. This "availability" is modified by the sales module, such that when a contract is started, the availability is removed (even if the availability was greater than the amount consumed in the contract -- something we probably should optimise later) and when the contract has finished, the availability is added back. In either case, once availability is added by the host (via rest api), perhaps the sales module should inform the datastore to "reserve" this space to not be consumed? What do you think @Bulat-Ziganshin?

first, probably these new REST APIs should be documented. It's up to you, just a side thought.
Check Dmitry's post before mine: https://github.com/status-im/nim-codex/issues/159#issuecomment-1205915683
He proposed to implement a very simple approach - we configure maximum disk usage by BlockStpore, and once the quota is reached, we just stop writing to the store. The obvious idea behind that is just to allow a node operator to set a hard limit of disk usage by Codex. As he said, it's just a quick-and-dirty solution. And it was better to implement for the October release.
Your REST API is already more demanding:

it requires an ability to compute how much space we are already occupied - well, this feature is already required for Dmitry's approach, so we need to implement it anyway
it requires an ability to reserve space for contracts. It can't be implemented now because
a) we don't have specific calls to distinguish between the writing of a contracted block and caching of an external block
b) we can't distinguish between contracted and cached blocks inside BlockStore

I tried to discuss here the extended variant of feature with Mark - before Dmitry said that we need to implement just the basic one. My thought - let's implement the basic one for the starter, its implementation is anyway first half of the implementation of the extended feature, plus it has more chances to become implemented prior to October release.

Understood. Keeping it simple for now for phase 1 with iterations later sounds like a very reasonable approach.
Perhaps this leaves us with two options:

Your initial suggestion of creating a datastore API call for unusedDiskQuota(), which could be called when a host is attempting to reserve available space via the rest api call.
Create an datastore API call that asks to reserve space, however at this point it would simply return a bool for whether or not there is currently sufficient space.

In my opinion, option 1 is the simplest for now, however option 2 is most similar to how the flow should be later on.

@emizzle it's easy to implement the first one.
The second one can be easily implemented via unusedDiskQuota(), BUT between the request and actual use of this storage the node may store some blocks from external sources, thus using extra space. And this means that we need to distinguish between external blocks just cached here, and paid space, i.e. we need to add extra parameter to putBlock. And this change will be also required to implement more advanced solution what can delete some blocks to reclaim space.
So, I propose

start with the implementation of unusedDiskQuota() and stupid reserveSpace() that just checks disk quota
continue with adding isPaid parameter to putBlock and corresponding modification in reserveSpace() implementation

and probably, it would be better to combine all changes in APIs called by Contracts together so that you can make all modifications on your side in a single PR

it would be better to combine all changes in APIs called by Contracts together so that you can make all modifications on your side in a single PR

Sounds good. I actually don't think any contract changes are required, it would only be a call to the "stupid" reserveSpace API call in datastore from the REST api call in nim-codex. But yes, I'm happy to put all the needed changes (won't be much) in a single PR once that API call has been implemented (or at least on a branch that I can branch off of).

I tried to find how contracts use putBlocks and didn't find anything - in the entire program putBlocks is called in 3 places - erasure.nim, block exchange for blocks received from network, and node.store() called by upload REST API

That's correct. Contract interaction with the blockstore is currently limited to the onStore callback: https://github.com/status-im/nim-codex/blob/e0726cbfb9b22dddf96c09f7523a3c714280fb37/codex/node.nim#L324-L345
This calls getBlock, which requests the blocks from the block exchange. Once resolved, these blocks will get put into the local store via putBlock: https://github.com/status-im/nim-codex/blob/e0726cbfb9b22dddf96c09f7523a3c714280fb37/codex/blockexchange/engine/engine.nim#L247-L257

perhaps the sales module should inform the datastore to "reserve" this space to not be consumed?

I don't think this is the correct approach. The functionality I have in mind is as follows - the user sets a quota that the node is allowed to consume, the node will return how much of that quota is still available for sale.
Also, we should not conflate caching blocks with persisting blocks, they might use the same underlying machinery (i.e. repo abstraction), but they should be configurable separately.
The difference between the two is:

caching quota will be used when downloading/uploading data, while persistent quota will be advertised for sale
The maximum amount of storage that the node is allowed to use is caching quota + persistent quota
The maximum amount allowed to be advertised for sale is persistent quota - used persistent quota
The maximum dataset size that can be uploaded is equal to caching quota

We might extend this to stream blocks to persisting nodes on the fly, but this isn't something we should worry about right now

The maximum dataset size that can be downloaded is equal to caching quota

We might extend this to allow streaming blocks and bypassing the caching store altogether, but this isn't something we should worry about right now

What I'm proposing as a stopgap solution, is to implement the basic caching quota functionality. What the repo should be able to do as a first step is:

report total amount of bytes used
error out if we try to add blocks past the quota

To support (1) and avoid lengthy scans of the repo to calculate the total used byte, we should keep track of how many bytes have been written and/or evicted to/from it. This can be recorded under /total/cachebytes and /total/persistentbytes (or similar).

Full functionality extensions
For the most part, both the caching and persistent quotas can be supported by the same underlying implementation. My current thinking is that we keep some metadata per block to support caching and persisting blocks. We need to keep track of the last time the block was accessed so we can evict it from the caching store and we need a flag that indicates wether the block should be counted as part of the caching or persistent quota.
The algorithm for storing blocks in the repo looks something like the following:

When storing a block
1.1. If the block should be persisted, we set it's persistent flag to true
1.2 We set it's last accessed field to the current time
1.3 If the block is persistent we count it against the /total/persistentbytes otherwise we count it against the /total/cachebytes
When we've reached the caching quota limit
2.1 If the block's persistent flag is not set, we'll evict the blocks with the oldest last accessed timestamp
2.2 If the block's persistent flag is set, we do nothing
2.3 If the caching quota has been reached and no blocks can be evicted (all blocks are persistent), we should not accept any more blocks - i.e. return error
Once all storage contracts that reference a particular block have finished
4.1. Set it's persistent flag to false
4.2. Subtract the total bytes from /total/persistentbytes and add them to /total/cachebytes
4.3. If caching quota has been reached, kick in the cleanup process as per (3)

The functionality I have in mind is as follows - the user sets a quota that the node is allowed to consume, the node will return how much of that quota is still available for sale.

Makes sense. We can update the rest api endpoint to check if there's enough persistent quota available in the datastore.

The maximum amount allowed to be advertised for sale is persistent quota - used persistent quota

@dryajov, will there be a datastore api that allows us to check the amount of persistent storage remaining?

For the FSStore DataStore it is pretty easy to reserve space by creating a RESERVE file of 100 GiB (or whatever).  Then you shrink that file as you add real chunks and grow it as you purge and we can ensure we can always honor the contract.  The reserve file size makes implementing quota-used just getFileSize.
This seems like a simple, serviceable & OS portable quota system.  Most of us and even many non-expert users have a lot of experience filling filesystems to 100% and then recovering usable space by deleting/truncating files - successfully even if there are not competing writers getting intermittent ENOSPCes.  So, I think this, too, could work and even with users pushing things to the brink and returning from said brink.
{ Some, but maybe not all subtleties: For our code we can block writes, but the FS can fill after we shrink RESERVE but before we use it, but selling against an 99+% full FS with alien writers is insoluble.  To reserve space, we need to either posix_fallocate or to write the file with data to pre-allocate it rather than just make a "hole".  Data written to a de-duplicating FS must be random; writing at all must be optional to not unduly wear out flash memory in testing. There might also need to be a little extra data freed up from the reserve for directory files. }
For SQLiteStore, big questions arise some of which @michaelsbradleyjr starts.."Does vacuum work if the host FS is 100% full, when compaction matters most - if not are you just stuck?", "If it works, what is the cost?", "Does DELETE,INSERT in the same table (or any other reservation idea) work as hoped with BLOB types of varying chunk sizes?"  Maybe more.
(A user with admin-ish rights on Unix can make an "FS within a file" to at least keep codex from crashing unluckily partitioned systems with disk full errors.  Maybe worth mentioning to early adopters as a way to not trust whatever we do here.  Apologies if this is old advice..It also has application in terms of our testing how well things work as you fill the host FS.)

Initial implementation, related to limits has been done in #319; subsequent work for repo maintenance is being carried out in #347.

I was thinking that for the purpose of deciding when to delete blocks, there could be no difference between cached and contract blocks: Cached blocks are kept for a short time in accordance with the caching settings. Contract blocks are kept for a longer but still limited time in accordance with the contract. I was thinking the expiration datetime for a contract block should be the duration of the contract + some grace period (hours?, days?, configurable).
I am unsure about whether the cached blocks and contract blocks can be treated identically when it comes to quotas. Do we want a max quota for the entire node, or one quota for caching and another for contract storage? Additionally, I like the idea of marking a block as non-persistent when the contract has expired. Currently, we delete blocks as soon as their expiration date has been reached. Perhaps we could consider a strategy of "don't delete unless you have to", where we fill up the quota on purpose and start cleaning the oldest, least interesting, non-persistent (any more) blocks only when we have no other way to respect the quota? Downsides: Write performance goes down because we may need to quickly make some room. It becomes trickier to answer the question how much space is available.

Some of the questions raised in this issue have already been addressed by the repostore implementation.

Cached blocks are kept for a short time in accordance with the caching settings. Contract blocks are kept for a longer but still limited time in accordance with the contract.

This is have it works already, the repostore treats all blocks equally and only relies on expiration to evict from the store.

I was thinking the expiration datetime for a contract block should be the duration of the contract + some grace period (hours?, days?, configurable).

This might be a nice addition, a grace period certainly makes sense in some circumstances...

I am unsure about whether the cached blocks and contract blocks can be treated identically when it comes to quotas.

The repostore has a notion of reserved storage and used storage. The marketplace can "reserve" the required amount before announcing/bidding on requests and release is right before writing blocks. This is a super simple but powerful mechanism that should cover all foreseeable use cases.
The flow is super simple, there is a reserve call which will reserve a number of bytes counted against all the used quota bytes - it should be called right before bidding on storage requests, and there is a release call - which should be called right before storing a block (not that I said block not blocks - this to prevent concurrency issues, but if that becomes problematic we can always add some synchronization to it).


Publish 8.4-20230905234741+0000

Publish 8.4-20230905234741+0000

@bot-gradle test and merge

I've triggered a build for you. Click here to see all failures if there's any.


CREATE A VENUE PAGE

Points to Note :

The issues will be assigned on a first come first serve basis, 1 Issue == 1 PR.
"Issue Title" and "PR Title should be the same. Include issue number along with it.
Follow Contributing Guidelines & Code of Conduct before start Contributing.
This issue is only for 'Hacktoberfest' contributors of 'FIFA WORLD CUP 2022 Project'.

:white_check_mark: To be Mentioned while taking the issue :

Full name :
GitHub Profile Link :
Participant ID (If not, then put NA) :
Approach for this Project :
What is your participant role? (Mention the Open Source Program name. Eg. HRSoC, GSSoC, GSOC, Hactoberfest etc.)

This page should contains all details of all the venues of FIFA WORLD CUP 2022.

i have done that page @Asttttha @Nirvik07

@patelyash5208 You need to first comment on an issue after that you will be assigned, then you can create PR. Then your PR will be merged.
This is according to the rule of contribution in Hacktoberfest and any Open Source Program.
And Assigning work as first comment first assigns . Please close your PR and wait for new issue or create your issue.

I have created issue please assign to me.
On Tue, 4 Oct, 2022, 10:36 am Nirvik Ghosh, @.***>
wrote:

@patelyash5208 https://github.com/patelyash5208 You need to first
comment on an issue after that you will be assigned, then you can create
PR. Then your PR will be merged.
This is according to the rule of contribution in Hacktoberfest and any
Open Source Program.
And Assigning work as first comment first assigns . Please close your PR
and wait for new issue or create your issue.
—
Reply to this email directly, view it on GitHub
https://github.com/Nirvik07/FIFA_WORLD_CUP_2022/issues/16#issuecomment-1266403221,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AT7Y6B2XF75DUIEP3HRJL7DWBO3NVANCNFSM6AAAAAAQ32PFJQ
.
You are receiving this because you were mentioned.Message ID:
@.***>


Any way to communicate request progress to swr?

Hello :)
Is there any plans to let us pass the progression percentage of the current fetch operation to useSWR?
Maybe something like
useSWR(
	'/big-dataset',
	(emitProgress) => /* fetch & call emitProgress when new data is loaded */,
	{ trackProgress: true },
);

This would allow us to use determinate progress indicators :)
Thanks!

I think one of the most important feature of SWR is keep it tiny but highly flexible (isolated from the actual fetch handler).
Related to #129.

That solution works for me, thanks :)

The issue with the proposition is that progress is scoped to the hook:
function useSWRProgress() {
  const [progress, emitProgress] = useState(0)
  const response = useSWR('/big-dataset', url => fetch(url, emitProgress))
  return {progress, ...response}
}

Using the hook in several places (or prefetch) will sync isLoading, isValidated, data, ... but progress will not.
The solution is to add a ProgressProvider which keep similar state to swr {[path]: progress} so the progress is globally shared, but this seems like a features which should be part of swr.


Accessible text for Edit links

Trello: https://trello.com/c/sYJYKnK9/58-how-summary-tables-handle-their-links-means-pages-can-contain-multiple-links-with-duplicate-text
If there are multiple links on a page with the same text, it can be confusing for screen reader users. This PR adds context in hidden text spans for the following links:

Editing an admin user (admin-manager role)
Uploading/downloading a supplier agreement file (admin-ccs-sourcing role)
Editing a supplier declaration (admin-ccs-sourcing role)
Editing a supplier service (admin-ccs-category role)
Viewing unapproved supplier service edits (admin-ccs-category role)

This should be reviewed by a content designer (just in case) as well as a dev.
Also fixes some single/double apostrophe clashes.
See previous PRs for Brief Responses and Supplier FEs:
https://github.com/alphagov/digitalmarketplace-brief-responses-frontend/32
https://github.com/alphagov/digitalmarketplace-supplier-frontend/pull/804

Agreed that the whole view_suppliers_agreements.html is a bit odd. It's hard to navigate as a sighted user, I can't imagine how annoying it must be for a screen reader user:

I will see if I can make the links a little less awful.

@katstevens oh wow, that is a one crowded page 😮.
Maybe we will be able to get rid of G-Cloud-7 at some point at least? 🤔
Anyway, thanks for taking time to improve the accessibility 💖.

The up and down arrows aren't part of the link and so it's difficult to sensibly put them in a summary.link, which also renders the table cell. Most of the other cells have rowspan=3 which is also not supported by summary.link. I think either this table has to get a full redesign, or the messy links stay 😿

I guess they stay for now then and we can highlight it to @laurenceberry and to whoever else it would be good to highlight it to? What do you think? Also if you need help let me know :).

We should flag it to whoever is going to have to update the Admin for G-10 😸 The poor CCS Sourcing team will have to wade through 6 columns instead of 5.

👍

Have added a tech debt ticket - this may (should?) get prioritised to a mission team dealing with G10 next quarter: https://trello.com/c/T2mJhHMp/389-admin-framework-agreement-management-page-will-need-to-fit-in-g-cloud-10-somehow


The path is displayed when you create a LineChart with a single point

Fix for an issue: #53

Hey Let me check that


fix(chart): Allow usage of helm built-in namespaces

This patch should resolve https://github.com/k8tz/k8tz/issues/5#issuecomment-950362169 using the existing ignore-namespace feature to resolve both. True, it's not backwards compatible, and if required, we can achieve this, but it should allow to use this helm chart, without requiring the chart to be used to create the namespace.
BREAKING CHANGE: Drop in-chart namespace creation
(One of the main reason to implement this change for me, is that I don't leave namespace creation to helm charts due to enforcing PSS and alike.)

I also built a fully backwards compatible version, but I think it just introduced unnecessary complexity: https://github.com/SISheogorath/k8tz/tree/fix/namespace-issues-compatible

Hi @SISheogorath,
sorry for the late reply.
we can't just break compatibility like that, unfortunately it is a feature from now on.
I did like most of your backward compatible approach, but there is a slightly nicer way imo. I started working on that in #92.
Please help to check and review if you can.
Thanks!


Docker-based install process requires people to clone repositories

Most people expect to just pull a published image and have the requirements instead the image.
Do people really need to clone all the repos just to try a tutorial?
It'd be nicer to just run a docker-compose.yml, which refers to images on docker hub.

Yeah, agreed. The plan is for that to happen once we have releases, so we can just use latest release images.


"no-multiple-empty-lines" issues at end-of-file are not shown

Hello,
I have the following rule in my .eslintrc file: "no-multiple-empty-lines": [2, {"max": 1}]
When running eslint from the command line, I see errors like:

error  Multiple blank lines not allowed                no-multiple-empty-lines

However, these issues are not shown in the IntelliJ IDEA UI, with the eslint-plugin, when the issue is at the end-of-file. It works otherwise (when the issue is before the end-of-file) though.
Cheers

Since the errors are marked by a squiggly red underline by default, if the line has no content, it is not visible.
You can, however, see in the right gutter, change the marking settings or check the 'Treat all eslint issues as warnings' in the settings page.

Hello,
Sorry, for some reason I don't have this problem anymore.
Thanks


error: undefined method `keys' for #Array:0x000000028643c0

I'm trying to convert a facebook feed json file, but it keeps returning this error:
error: undefined method `keys' for #Array:0x000000028643c0

I didn't attempt to patch this gem but I did post a small script that handles arrays in json to csv conversion here:
http://stackoverflow.com/questions/7845015/convert-json-to-csv-in-ruby


OpenSSL Crypto error when

I'm using the latest key.
OpenSSL.crypto.Error: [('PEM routines', 'get_name', 'no start line')]
[1/Jan/2020 12:32:55] "GET /admin/analytics/token/2/ HTTP/1.1" 500 17912

This appears to be a duplicate of #25 and may have been solved in #44.


Testing - Changing from 3.* python to 2.7

Refs: #16962

[ ] macOS @jrieken
[ ] linux @connor4312
[ ] macOS @chrisdias (Optional)
[ ] windows @tanhakabir

Complexity: 3
Create Issue

Requirements

You will need two versions of python for this 3.* and 2.7.

You can either install from python.org
OR
You can use pyenv mac/linux or pyenv-win windows. Then use pyenv install 3.9.4 2.7.18 to get the two verisons.

Install python extension from here: https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix
Make sure you have "python.experiments.optInto": ["All"], in your user settings. If you did not have it, add it and reload.

Testing

Open an empty folder in VS Code.
Add following unittest code:

import unittest
class PassingTests(unittest.TestCase):
    def test_passing(self):
        self.assertEqual(42, 42)
    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")
class FailingTests(unittest.TestCase):
    def test_failure(self):
        self.assertEqual(42, -13)
    def test_failure_still(self):
        self.assertEqual("I'm right!", "no, I am!")

Start with python version 3.*.
You can either configure tests using Python: configure tests... command from the command pallet, or go the test view and configure by using the welcome view, or you can run tests to get a prompt to configure.
Select unittest framework.
You should see the tests in the test view. Run the tests.
Now change python version to python 2.7, it should automatically refresh and allow you to run tests with python 2.7

Kind of blocked because test discovery doesn't work for me

@jrieken Can you elaborate on where you are stuck?
If you have the test file in the root of the project then your settings should look like this:
{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        ".",
        "-p",
        "*test*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

If your test file is in a sub-directory (say mytests) then it should look like this:
{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./mytests",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

Can you share how the settings looks like?

@jrieken is it just the file name issue? The instructions originally had a file name suggestion, but I just updated them to flat-out specify a file name. Otherwise is there something else you consider too ambiguous for non-Python users?

Specifying the file name to use helped a lot, because

I've ran into the issue of accidentally overriding a module before while testing out a Python feature
It corresponds to one of the entries in the quick pick while running the configure command.

Still, not working for me
https://user-images.githubusercontent.com/1794099/130751466-1c9d493a-c562-4d57-a154-e50a070a0391.mov


WMO Region 7 Antarctica

WMO Region 7 Antarctica cannot be added to station descriptions

The next version of the wis2box will use the code list directly from https://codes.wmo.int/wmdr/_WMORegion, this should resolve the issue.

@kornada the latest version of the station editing form now uses the code list from codes.wmo.int, including Antarctica. Please see the screen shot below and the current main branch.

For you second comment, on the traditional station identifier, please note that the purpose of this field in the station metadata form is to allow linking between the identifiers in BUFR (typically the block and station number, ship call sign or buoy number) and the WIGOS station identifier. The form is not intended to replace or replicate the contents of OSCAR/Surface.

Closing, @kornada please open another issue if required.


Add alchemiscale.org-specific environment files, Docker image build automation

The OpenFreeEnergy/alchemiscale repo features Docker image builds based on environment files maintained in the repo, updated with new releases. These "batteries-included" images feature openfe, feflow, and other OpenFE packages that are of general interest for any alchemiscale deployment.
However, for alchemiscale.org, we want deployment-specific images that additionally feature things like alchemiscale-fah that are not of general interest, or even experimental protocol packages such as pontibus. These images should be built from environment files maintained in this repo.
We will build these images as ghcr.io/openfreeenergy/alchemiscale.org-server and ghcr.ioopenfreeenergy/alchemiscale.org-compute, for server and compute respectively.

Completed in first commits to main! See 2c8faad996b8e402bd20ca7e1ddb0c46e3738eb0 for most of this.


RTS: remove a TODO

Redundant #![feature(...)]s generate warnings as we update rustc, so
no need to add a TODO about removing them.

CI segfaulted, again.
3 of my recent PRs with comments-only changes all blocked with segfaults in Darwin CI.


PasswordCredentialPanel: handle authentication rejection

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

7 of 7 (100.0%)  changed or added relevant lines in 1 file are covered.
9 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.08%) to 99.403%

Files with Coverage Reduction
New Missed Lines
%

src/modules/olMap/handler/measure/OlMeasurementHandler.js
1
96.49%

src/modules/olMap/handler/draw/OlDrawHandler.js
8
96.94%

Totals

Change from base Build<PHONE_NUMBER>:
-0.08%

Covered Lines:
10066

Relevant Lines:
10077

💛  - Coveralls


Getting buffer progress in chunks

On the hlsjs demo page, buffering is shown in chunks, like this:

I want to implement this in clappr, but when looking at player.progress  I only get an object with: { start, current, total }. I would expect an array, looking something like this: [ { start, end}, { start, end }, ... ]. Am I looking at the wrong place or is this not currently possible?

@tobiasandersen currently this is not possible, the progress event filters the buffer containing the current position.
I'll mark this as a enhancement, so we can think of a better way to provide these buffer positions.

@tobiasandersen 2f7fc506ea683b5d79ee23201561a5658597285d and 8648f25bffad2983fa62a4702b827d4943f7b9ac added a second parameter to HTML5 based-playbacks to include an array of these buffered segments.
You can now use do something like this:
player.getCurrentPlayback().on(Clappr.Events.PLAYBACK_PROGRESS, function(progress, buffered) {
  // progress is the same object to keep compatibility: { start, current, total }
  // buffered is an array in the format: [{start, end}, {start, end}, ...]
}

Great, thank you!


Smart contracts 0.8.0 release

Problem
New smart contracts and features need to be captured in a new release
Solution

[x] Create a release/0.8 branch from the latest commit in main.
[x] Create a new branch named bump-0.8.0-rc1. Submit a 0.8.0-rc1 bump PR from the bump-0.8.0-rc1 branch to the release/0.8 branch. This PR should only update the version statement in package.json and package-lock.json to 0.8.0-rc1. (Example: https://github.com/hashgraph/hedera-smart-contracts/pull/631)
[x] Create a new branch named bump-0.9.0-snapshot. Submit a 0.9.0-SNAPSHOT bump PR from the bump-0.9.0-snapshot branch to the main branch. This PR should only update the version statement in package.json and package-lock.json to 0.9.0-SNAPSHOT. (Example: https://github.com/hashgraph/hedera-smart-contracts/pull/632)
[x] Establish a new release with the title v0.8.0-rc1, tageting release/0.8 branch, and set it as pre-release. Simultaneously create a corresponding tag with the same title for the release. Publish the release. (Example: v0.7.0-rc1 release)
[x] Establish a new release with the title v0.8.0, targeting release/0.8 branch, and set it as latest release.Simultaneously create a corresponding tag with the same title and compose the release note. Publish the release. (Example: v0.7.0 release)

Alternatives
No response

https://github.com/hashgraph/hedera-smart-contracts/releases/tag/v0.8.0


-dump-ast overrides -wmo and -index-file

This fixes the Driver/ast_dump_with_WMO.swift test, and expands the override to cover -index-file as well based on a FIXME in the C++ source. If both are specified, a warning is emitted: ignoring '-wmo' because '-dump-ast' was also specified

@swift-ci please test


Ticking screen crash

Issue description:
Having a ticking screen crash occur on a server, it doesn't happen to everyone so must be client side I'm guessing. Running on the Foolcraft pack 1.3.2. The grids have worked previously for me and are still working for some others on the server. Have no idea what initially started the problem.
What happens:
Crash to desktop when opening any grid/wireless grid. Gui appears briefly, doesn't load any of the items
What you expected to happen:
The grid gui to open properly
Steps to reproduce:

Opening any grid

Version (Make sure you are on the latest version before reporting):

Minecraft: 1.10.2
Forge:
Refined Storage: 1.2.25

Does this issue occur on a server? [yes/no]
yes
If a (crash)log is relevant for this issue, link it here:
[https://puu.sh/vOZsY/01b1bb8b09.txt]

The crash happens while it's trying to get the display name of an item in your grid.
The problem is at the item side, but unfortunately the crash tells nothing about which mod that causes this.

Located the disk drive manually, dumping the items into a chest. This seems to be a common issue on our server so I'll post back once I find the item.

Managed to export the item into a chest, it just doesn't appear so I'm guess the best fix for this at the moment is just voiding the drive.

Okay, I'll have to close this issue then, since it isn't RS' fault.


isAnimating is not working with AVIFS

I am trying to make this example work. It does work with mp4 on my end. However, it does not work with AVIFS. Since I am loading many AVIFS files within a LazyHStack, I am getting memory issues, therefore I need to be able to stop them.
     VStack {
                                            AnimatedImage(url: URL(string:"https://raw.githubusercontent.com/link-u/avif-sample-images/master/star-12bpc-with-alpha.avifs")!, isAnimating: $isAnimating)
                                        Button(action: {
                                            self.isAnimating.toggle()

                                        }, label: {
                                            HStack{
                                                Text("Stop")
                                                    .font(.system(.title3))
                                                    .fontWeight(.bold)

                                            }
                                        })
                                        }

...I guess this is similiar to your previous issues report.
The AnimatedImage actually does not care about any Image Format
If it's a bug of isAnimating animtion control, then all animated image must encounter this bug.

I can not reproduce on our demo, can you provide the full reproducable project or full source code to analayze ?

I think this is still something about your Library version or integration issue
Check https://github.com/SDWebImage/SDWebImageSwiftUI/pull/292 again

SDWebImage: use latest
SDWebImageSwiftUI: use 2.x latest or 3.0.0-beta2
libavif: use latest and choose dav1d as codec (subspecs => 'libdav1d')
libdav1d: use latest


Add bulk_published to aaib-report example

This is following a conversation on specialist-frontend bugfix PR.
cc @tommyp @boffbowsh

Nice work @binaryberry! 👍

Thanks for all your help @tommyp  !


Bootstrap Dependencies

Adding bootstrap dependencies to the project

Bootstrap 3.3.6
jQuery 2.1.4

Merging with master from branch frontend.

NOTE: The extra html files is static and temporary. Will be modified later.


update readme.md for better understanding of concept

update readme.md for better understanding of the concept. Add a table of contents or anything you like that me it more understandable

Hello @Abhijay007 ,
I would like to work on this issue,
Kindly assign this to me & I will begin working on it right away!

I have updated the readme file, can I make the pull request?

I have updated the readme file, can I make the pull request?

Yeah sure!!

Hello @Abhijay007
I have created a Pull Request, Kindly Check it out and let me know.
If you think it is good then Merge .
Cheers,
Ck Cheema

@Abhijay007
I have made the changes and created a pull request. Please have a look and let me know if you want any further changes

issue resolved

issue resolved


Edit mode - In sentence-by-sentence option disable "Add new translation" button.

Caused because of ui changes, and fixed by #76.


Closing the project should gracefully shutdown all services

This task is automatically imported from the old Task Issue Board and it was originally created by Hubert Plociniczak.
Original issue is here.

[info] [2023-01-30T12:17:32.379Z] [enso.org.enso.compiler.PackageRepository$Default] Found library Standard.Database @ 2023.1.1-dev at [***/2023.1.1-dev].
[error] [2023-01-30T12:17:44.168Z] [org.enso.languageserver.runtime.ContextEventsListener] Unable to find suggestion for [(Standard.Base.Data,Standard.Base.Data,fetch)].
[error] [2023-01-30T12:17:45.198Z] [org.enso.languageserver.runtime.ContextEventsListener] Unable to find suggestion for [(Standard.Table.Data.Table_Conversions,Standard.Table.Data.Table.Table.type,from_objects)].
[info] [2023-01-30T12:19:31.914Z] [org.enso.projectmanager.protocol.ClientController] Client disconnected from the Project Manager [8ef19391-b4b2-4dcb-8ae8-907b4b3d059a]
[info] [2023-01-30T12:19:31.914Z] [org.enso.languageserver.protocol.binary.BinaryConnectionController] Connection closed [<IP_ADDRESS>:51052].
[error] [2023-01-30T12:19:31.93Z] [org.enso.projectmanager.boot.ProjectManager$] Main process execution failed.
java.io.EOFException: Some(There is no more input left to read)
  at zio.console.package$Console$Service$$anon$1.$anonfun$getStrLn$1(package.scala:60)
  at zio.internal.FiberContext.evaluateNow(FiberContext.scala:490)
  at zio.internal.FiberContext.$anonfun$fork$17(FiberContext.scala:770)
  at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base@11.0.17/java.lang.Thread.run(Thread.java:829)
  at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
  at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)

Closed by #7157


failed to get trust policy: no policy found for reference when scope matching with image:tag

What happened in your environment?
Following this doc, https://github.com/deislabs/ratify-web/blob/e0d548665d273502be477559d10fc02911348c51/docs/plugins/Verifier/cosign.md#trust-policy

I specified a image:tab as the scope of the cosign trust policy, however since the image to deploy has mutated to the digest, verifier was not able to find a trust policy that matches image:tag.
Error: Detail: failed to get trust policy: no policy found for reference
What did you expect to happen?
No response
What version of Kubernetes are you running?
No response
What version of Ratify are you running?
No response
Anything else you would like to add?
No response
Are you willing to submit PRs to contribute to this bug fix?

[ ] Yes, I am willing to implement it.

@akashsinghal , does the notation trust policy have the same limitation?
For now, we can update the doc to make sure Customer are redirected to use the digest, or specify wildcard for scope matching.

@akashsinghal , does the notation trust policy have the same limitation? For now, we can update the doc to make sure Customer are redirected to use the digest, or specify wildcard for scope matching.

In terms of notation spec on trust policy, the scope can be either * or a path to repo. So it will not have the issue parsing image tag to digest.

I've added a note in the documentation about this behavior. At this point, I think that's all that we will support.

Discussed at community , no plan to change this behavior right now.


fix: Allow dataset owners to see their datasets

SUMMARY

Allow dataset owners to view / access their own datasets.
TESTING INSTRUCTIONS

Detailed steps to reproduce are outlined in #19360.
ADDITIONAL INFORMATION

[x] Has associated issue: Fixes #19360
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Hi! By looking at the release notes I see this not being part of 2.0, any idea in which version this fix will be released? Thanks!

Hi! By looking at the release notes I see this not being part of 2.0, any idea in which version this fix will be released? Thanks!

@agusfigueroa-htg This fix was already included in version 1.5.1 so it's part of 2.0.0 too.

Amazing, thanks for the info Michael!
On Fri, Jul 15, 2022 at 1:47 PM Michael S. Molina @.***>
wrote:

Hi! By looking at the release notes I see this not being part of 2.0, any
idea in which version this fix will be released? Thanks!
@agusfigueroa-htg https://github.com/agusfigueroa-htg This fix was
already included in version 1.5.1 so it's part of 2.0.0 too.
—
Reply to this email directly, view it on GitHub
https://github.com/apache/superset/pull/20135#issuecomment-1185466579,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASNRLXX6LQHTV6HL6O4YJJDVUFFUPANCNFSM5WNRANVA
.
You are receiving this because you were mentioned.Message ID:
@.***>


[SPARK-35700][SQL][FOLLOWUP] Read schema from ORC files should strip CHAR/VARCHAR types

What changes were proposed in this pull request?

This is a followup of https://github.com/apache/spark/pull/33001 , to provide a more direct fix.
The regression in 3.1 was caused by the fact that we changed the parser and allow the parser to return CHAR/VARCHAR type. We should have replaced CHAR/VARCHAR with STRING before the data type flows into the query engine, however, OrcUtils is missed.
When reading ORC files, at the task side we will read the real schema from ORC file metadata, then apply filter pushdown. For some reason, the implementation turns ORC schema to Spark schema before filter pushdown, and this step does not strip CHAR/VARCHAR. Note, for Parquet we use the Parquet schema directly in filter pushdown, and do not this have problem.
This PR proposes to replace the CHAR/VARCHAR with STRING when turning ORC schema to Spark schema.
Why are the changes needed?

a more directly bug fix
Does this PR introduce any user-facing change?

no
How was this patch tested?

existing tests

@yaooqinn @dongjoon-hyun @HyukjinKwon

Test build #140159 has started for PR 33030 at commit 87124f2.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44685/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44685/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44685/

Merged to master/3.1.

Test build #140159 has finished for PR 33030 at commit 87124f2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140159/


Error: [ethjs-rpc] rpc error with payload when used on Kovan or Mainnet

ethjs-signer
Issue Type

[x] Bug (https://github.com/ethjs/ethjs-signer/blob/master/.github/CONTRIBUTING.md#bug-reports)
[ ] Feature (https://github.com/ethjs/ethjs-signer/blob/master/.github/CONTRIBUTING.md#feature-requests)

Description
An Error: [ethjs-rpc] rpc error with payload error is thrown when implementing the code from the usage instructions section of the README on both the Kovan and Main networks.
Error: [ethjs-rpc] rpc error with payload {"id":339528...,"jsonrpc":"2.0","params":["0xf86..."],"method":"eth_sendRawTransaction"} null
    at bundle.js:13279
    at XMLHttpRequest.request.onreadystatechange (bundle.js:8474)

This appears to be a ethjs-rpc error, but it is happening when signing transactions using the ethjs-signer package.
Steps to reproduce

Copy the usage code example from the README
Update the HttpProvider replacing the localhost URL new HttpProvider('http://localhost:8545') with the Kovan testnet or the mainnet URL for e.g. new HttpProvider('https://kovan.infura.io/AccessTokenHere')
Replace the account address and private key with details of a real account with an Eth balance
Webpack and run the code

Versions

Node/NPM: Node v6.9.1 / NPM 3.10.8
Browser: Chrome version 57.0.2987.133 (64-bit)
OS: macOS Sierra version 10.12.4 (16E195)

@mattvick will review, the code is simple enough if you find a fix just make a PR.

There were a couple of things I did that caused this:

I was not defining the gasPrice when creating my transactions, which meant they were not executed and became 'stuck'
When creating new transactions I was incrementing the nonce manually, meant that several 'stuck' transactions were created that needed to be overwritten

To assist others I've added a the missing gasPricedefinition to the usage example in the README.md and a note that this is required.
Credit goes @arkpar who pointed me in the right direction on Gitter


dissectors: hidpp: adds option to set initial HID++ version

This feels a little dirty, perhaps there's a better way to do this?

@Lekensteyn could you have a look at this? Do you think there's a better way?

I am going to merge this for the time being, @Lekensteyn please let me know if you have any suggestion.

This might not work properly in the following cases:

A receiver is paired with a HID++ 1.0 device (e.g. K800) and a HID++ 2.0 device (e.g. M525).
Multiple receivers are in use.

To solve the latter, I suggest including the device address in the key.
To solve the former, I suggest including the device index in the key.
While the C dissector API allows you to set "per-conversation" or "per-packet" metadata, this is not (yet?) possible in Lua. So using a global variable seems to be a reasonable workaround. You might want to clear the table in an "init routine" though, otherwise it'll also affect dissection of a different capture file. Something like this:
function hidpp_proto.init()
    hidpp_version_packets = {}
end

(In C there is a difference between a "cleanup" and "init routine", but in Lua the "init" function is called for both the "init routine" (called when a capture file is loaded) and a "cleanup routine" (called when a capture file is closed).

Okay, this seems pretty easy. If you don't mind I will @ once I make the changes :grin:.


Ability to Reference Variables in Styles when Created via the Plugin

Is your feature request related to a problem? Please describe.
Figma now supports the ability to reference variables in styles. Users would possibly want to be able to do this in the future but we currently don't support this.

Will be shipped in 2.0


Don't think @memorize fit any case.

I saw you use first argument as key of cache.
What if the first argument is an object, like this:
const a = {};
const b = {c: {}}
String(a) //"[object Object]"
String(b) //"[object Object]"

Besides, can you provide any benchmark tests?
I don't think it fit any case.

It only supports primitive values. This is the reason it is fast.


Allow .DISP to be changed from the ioc side

In pandablocks-ioc we'd like to set the DISP of some PVs so that they can't be changed during HDF5:Capture. The only way to set this with the current pandablocks-ioc is through caput. It'd be nice to be able to set DISP from the ioc side.

This should be relatively straightforward to implement.  I'm no longer set up to actually run pythonIoc anymore, so try the following untested patch:
diff --git a/softioc/device.py b/softioc/device.py
index 4fcb5ad..9f5820b 100644
--- a/softioc/device.py
+++ b/softioc/device.py
@@ -83,6 +83,12 @@ class ProcessDeviceSupportCore(DeviceSupportCore, RecordLookup):
     def _write_value(self, record, value):
         record.write_val(value)

+    def set_field(self, field, value):
+        data = (c_char * 40)()
+        data.value = str(value).encode() + b'\0'
+        db_put_field(
+            self._record.NAME + '.' + field, fields.DBF_STRING, data, 1)
+

 class ProcessDeviceSupportIn(ProcessDeviceSupportCore):
     _link_ = 'INP'


Confirmation token looks not valid on sign up

Hello,
I have a rails 4.0.4 app with devise 3.2.2 with confirmable enabled.
When a user sign up he receives by email the confirmation email with the link to be clicked to confirm user. Clicking on this link devise answers with "invalid token".
If the user requests confirmation email again, clicking on the link it works as expected.
I have not customized anything devise-related so I have no idea why this difference between "confirmation-on-sign-up" vs "confirmation-on-request"
Thank you!

We also encountered this problem. We did the following: Devise provides an after confirm hook called after_confirmation, which was sufficient for our needs, since we wanted to set some properties for the user when he uses them (after he confirmed his account successfully). Hence we moved the code from the after_create to the after_confirmation method and it worked out as intented.
Since this will not work for every usecase, since some of them want to set properties directly after creation and before confirmation in particular, but I hope this will help some of you.
Best regards

I just ran into this issue and created a brand new Rails app to test.  I found that if I use build in the after_create statement then I have the problem of the unconfirmed_email getting set.  My simple setup was as follows:
# app/models/user.rb
class User < ActiveRecord::Base
  # Include default devise modules. Others available are:
  # :confirmable, :lockable, :timeoutable and :omniauthable
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :trackable, :validatable, :confirmable

  has_many :projects
  after_create :add_project

  private

    def add_project
     # Using create DOES NOT set the unconfirmed_email and
     # thus I can confirm my user with no problem
     # self.projects.create(name: "foo")

     # Using build DOES set the unconfirmed_email and
     # thus my confirmation always fails.
     self.projects.build(name: "foo")
     self.save
    end
end

# app/models/project.rb
class Project < ActiveRecord::Base
  belongs_to :user
end

When you build a project the user model is affected, because of the relation to the project model. But the "problem" is the save method you call on the user model. Once you save the model the confirmation token gets regenerated and therefore every mail you sent with the (now) old token is useless.
Again we solved this by moving all the code to the after_confirmation method, which gets called once the user is confirmed. This might be also a solution for your situation, since I user is only allowed to work on projects after he is confirmed (I suppose). Therefore add the project onla after confirmation.

I understand that this may not be a bug, but it is very difficult to figure out why confirmable is not working in this case. Determining that the after_create on the user model is the problem feels somewhat obscure.  Googling the issue brings up posts regarding the switch from the secure to the insecure token which was not the problem.  I had to then go through the Devise gem itself until I realized that the postpone_email_change_until_confirmation_and_regenerate_confirmation_token was also being called along with generate_confirmation_token which finally led me to an SO post that highlighted that the after_create method was the problem.  Moving the code to the after_confirmation method is fine except when you want to allow your user to use the site immediately without confirming right away, or when you only want something to happen once and then you have to modify the after_confirmation method to deal with reconfirmation.  Is there absolutely no way around this?  At the very least can a note be added (I'm happy to do so) to the wiki post on adding confirmable that an after_create method on the user model may contribute to confirmable not working properly to save some people a couple hours of digging?  Or perhaps a new wiki page altogether outlining the issue and how you can move things to after_confirmation method and also handle reconfirmation a la this post.

:hurtrealbad:


libwmf: <IP_ADDRESS> -> 0.2.12

Motivation for this change
Includes the previous security fixes:
- CVE-2006-3376
- CVE-2009-1364
- CVE-2015-0848+4588+4695+4696
- CVE-2016-9011
+ fixes CVE-2019-6978 (in libgd)
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

nox-review builds for me, non-nixos linux x86_64

Thank you for the extensive test @risicle


Fix our text encoding

We are not encoding text correctly so there are cases where text can be copied into a text box that has symbols that our database cant accept causing it to throw errors.

 Fix our text encoding #776

 Fix the text encoding #776


AMD/UMD Support

Hi,
I'm trying to develop a boilerplate using this library but, I cannot require it in a view using requirejs.
As I can see from the dist/avoriaz.js file, this needs just 3 dependencies:

vue
vue-add-globals
lodash

...maybe this implementation will not be so hard, can you give it a try?
Ps. Thank you for the great job here man!

I've added umd and amd builds locally, but I'm not sure if they should include Vue. If they did, the dist files would be tied to whichever version of Vue avoriaz is using. I think the user should be able to choose which version they run.
I don't use umd/amd files often. Can you tell me a bit about your test pipeline. do you have an example project I can look at? Would you be able to include Vue seperately from avoriaz?
I think avoriaz should have avoriaz.umd and avoriaz.amd files that include lodash/cloneDeep and vue-add-globals, but not Vue.
What do you think?

Totally agree with the inclusion of lodash/cloneDeep and vue-add-globals and the exclusion of Vue, every dev should be free to choose the Vue's version in every kind of environment...just think about the .min.js (production) and max version (dev with suggestions in console).
Actually i'm not ready to publish the boilerplate but, this is the project's configuration:

Karma as test runner
PhantomJS as browser
Mocha, Chai, Avoriaz as tests library

About karma main (client) file
When test browser will be executed, this file will be injected.
This is the very entry point of the test environment, the reason why we put here the first require.config call, configuring the modules alias (require.config >> paths) to all the tests files + test library (chai, avoriaz).
Tests boot

Test browser launcher executed
Karma inject plugins mocha and requirejs
Karma inject configurations in globals
Karma main file executed, first require.config done
All the tests files will be loaded and executed (via require, AMD).

Test files
Every test file will be wrapped in define (requirejs) function as follow:
define(["vue", "chai", "avoriaz", "/base/src/mycomponent"], function(Vue, chai, avoriaz, component) {
	// test assertions on component...
});

Ok, I've released a umd and amd build in 3.5.0
You can require with 'avoriaz/dist/avoriaz.amd.js' or 'avoriaz/dist/avoriaz.umd.js'
Can you let me know if this works correctly for you?

Awesome!
It's working like a boss...i tried with both amd and umd version.
I was thinking about it, do we need really 2 files? amd and umd?
Maybe the umd is enought to cover every needs, what do you think about it?

Glad it's working ☺.
It's not much extra effort to have a umd and amd file, so I think we can
leave it.
On 25 Aug 2017 21:16, "Micene09"<EMAIL_ADDRESS>wrote:
Awesome!
It's working like a boss...i tried with both amd and umd version.
I was thinking about it, do we need really 2 files? amd and umd?
Maybe the umd is enought to cover every needs, what do you think about it?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/eddyerburgh/avoriaz/issues/119#issuecomment-325024606,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMlbWwFsBn0zQxFlmI5VK2BLhMk-sU7iks5sbyungaJpZM4PAWhb
.

Ok man.
Thank you for the great job here.
Closing.


refactor: misc

Codecov Report
Base: 100.00% // Head: 99.91% // Decreases project coverage by -0.09% :warning:

Coverage data is based on head (7e50b35) compared to base (bab863d).
Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head 7e50b35 differs from pull request most recent head 3b4d7c5. Consider uploading reports for the commit 3b4d7c5 to get more accurate results

Additional details and impacted files
@@             Coverage Diff             @@
##              main      #19      +/-   ##
===========================================
- Coverage   100.00%   99.91%   -0.09%
===========================================
  Files           48       48
  Lines         2238     2238
===========================================
- Hits          2238     2236       -2
- Misses           0        2       +2

Impacted Files
Coverage Δ

pkg/context/context.go
100.00% <100.00%> (ø)

pkg/crypto/x509/x509.go
100.00% <100.00%> (ø)

pkg/env/env.go
100.00% <100.00%> (ø)

pkg/jose/header.go
100.00% <100.00%> (ø)

pkg/jose/jwa/jws.go
100.00% <100.00%> (ø)

pkg/jose/jwa/sign.go
100.00% <100.00%> (ø)

pkg/jose/jwa/verify.go
100.00% <100.00%> (ø)

pkg/jose/jwk/jwk.go
100.00% <100.00%> (ø)

pkg/jose/jws/verify.go
100.00% <100.00%> (ø)

pkg/jose/jwt/claims.go
100.00% <100.00%> (ø)

... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Getting a invalid URL message when we try to access the magic link through the received email

Describe the issue:

Getting a invalid URL message when we try to access the magic link through the received email

https://github.com/wso2/product-is/assets/40081059/d60aef61-04bf-4c1d-88c3-cda371b6b546
How to reproduce:

1.Go to the console
2.Create and application
3.Set up basic as 1 st factor  magic link as the 2nd factor  as the sign in method.
4.Go to the login flow ,  enter usrname and pws and complete the password
5.Click on continue
6.Go to the received email and click on the  "Sign in" button
Expected behavior:

Environment information (Please complete the following information; remove  any unnecessary fields) :

Product Version: [e.g., IS 5.10.0, IS 5.9.0]
OS: [e.g., Windows, Linux, Mac]
Database: [e.g., MySQL, H2]
Userstore: [e.g., LDAP, JDBC]

Optional Fields
Related issues:

Suggested labels:

https://github.com/wso2-extensions/identity-event-handler-notification/pull/209


Migrate Blueprint Popover to Mui Popover

To continue the effort of removing dependency on Blueprint, let's replace Blueprint Popover to MUI Popover!

This is still needed!

@tihuan @seve Is this easy enough to implement that it could be considered an on-call issue for the DP team?

Sorry just saw this! I just updated the ticket to migrate to MUI Popover instead of Blueprint Popover2, since we are trying to get rid of Blueprint in DP
I think this can be done as an oncall issue 👍 Thank you!


Admin user could not roll back ingestion when REST_API_AUTHORIZATION_ENABLED is set to true

Describe the bug
In the latest release v0.11.0 with env variable REST_API_AUTHORIZATION_ENABLED is set to true. The ingestion roll back seems failing and the GMS log shows that the user is not authorized to delete entities, even though we are logged in as datahub root user who didn’t touch anything on permissions
To Reproduce
Steps to reproduce the behavior:

Locally deploy a latest Datahub release with env variable REST_API_AUTHORIZATION_ENABLED set to true
log in as datahub user
Run a sample ingestion which will produce some metadata
Roll back ingestion, you'll see the failure

Expected behavior
User with admin role or root user should be able to roll back ingestion
Additional context
GMS log is saying user is unauthorized to delete entities.
Caused by: com.linkedin.restli.server.RestLiServiceException: User is unauthorized to delete entities.
	at com.linkedin.metadata.resources.entity.BatchIngestionRunResource.lambda$rollback$7(BatchIngestionRunResource.java:133)
	at com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:30)
	at com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)
	at com.linkedin.metadata.resources.entity.BatchIngestionRunResource.rollback(BatchIngestionRunResource.java:112)

This is resolved by the #9092


fetch mission data

Fetch mission data from API

Hi @monicah79 ,
STATUS: APPROVED 😃
git-merge

Your project is complete! There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉
Highlights
All changes requested have been completed ✔
Optional suggestions
Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.


Add debian bookworm support

This PR bumps the puppet_agent module version to include support for Debian Bookworm.
This fixes #7
This was generated by updating the puppet_agent submodule and running ruby install.rb.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@span786 Were you about to look into this?

Changes for debian-12-x86-64 were taken care in https://github.com/puppetlabs/install-puppet/pull/20

@linuxdaemon could you update your PR to include the litmus and Perl changes?

I think this is blocking support for Debian 12 in puppetlabs modules, like https://github.com/puppetlabs/puppetlabs-apt/pull/1178 for example.

Hi @linuxdaemon Thanks for updating the install-puppet.sh script to match what's in the puppet_agent module, but I'm not following the part about "adds bookworm support". Wasn't that handled https://github.com/puppetlabs/install-puppet/pull/20/files#diff-043df5bdbf6639d7a77e1d44c5226fd7371e5259a1e4df3a0dd5d64c30dca44fR871?
Separately, I think there is a issue with https://apt.puppet.com/puppet-release-bookworm.deb that install-puppet.sh downloads by default. It seems to install an old version of puppet-agent from debian's repos(!)
$ docker run -it debian:bookworm /bin/bash
root@e3f41b5a291d:/# apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8786 kB]
Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [13.8 kB]
Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [156 kB]
Fetched 9210 kB in 1s (11.2 MB/s)
Reading package lists... Done
root@e3f41b5a291d:/# apt install curl
...
root@e3f41b5a291d:/# curl -LO https://raw.githubusercontent.com/puppetlabs/install-puppet/619cedb37831114f6499a429e473296925c8b40c/install.sh
...
root@e3f41b5a291d:/# bash install.sh
05:21:40 +0000 INFO: Version parameter not defined and no agent detected. Assuming latest.
05:21:40 +0000 INFO: Downloading Puppet latest for Debian...
05:21:40 +0000 INFO: Debian platform! Lets get you a DEB...
05:21:40 +0000 INFO: Downloading http://apt.puppet.com/puppet-release-bookworm.deb
05:21:40 +0000 INFO:   to file /tmp/install.sh.2841.924572179/puppet-release-bookworm.deb
05:21:40 +0000 INFO: Trying curl...
05:21:41 +0000 INFO: Installing puppetlabs apt repo with dpkg...
Selecting previously unselected package puppet-release.
(Reading database ... 6745 files and directories currently installed.)
Preparing to unpack .../puppet-release-bookworm.deb ...
Unpacking puppet-release (1.0.0-31bookworm) ...
Setting up puppet-release (1.0.0-31bookworm) ...
Get:1 http://apt.puppet.com bookworm InRelease [56.2 kB]
Get:2 http://apt.puppet.com bookworm/puppet all Packages [517 B]
Get:3 http://apt.puppet.com bookworm/puppet amd64 Packages [517 B]
Hit:4 http://deb.debian.org/debian bookworm InRelease
Hit:5 http://deb.debian.org/debian bookworm-updates InRelease
Hit:6 http://deb.debian.org/debian-security bookworm-security InRelease
...
Get:32 http://deb.debian.org/debian bookworm/main amd64 puppet-agent all 7.23.0-1 [1147 kB]
root@e3f41b5a291d:/# which puppet
/usr/bin/puppet
root@e3f41b5a291d:/# puppet --version
7.23.0

If I manually install the puppet7-release-bookworm.deb package then it works:
$ docker run -it -v `pwd`:/root debian:bookworm /bin/bash
root@a9f00190986d:/# dpkg -i /root/puppet7-release-bookworm.deb
Selecting previously unselected package puppet7-release.
(Reading database ... 6090 files and directories currently installed.)
Preparing to unpack .../puppet7-release-bookworm.deb ...
Unpacking puppet7-release (7.0.0-17bookworm) ...
Setting up puppet7-release (7.0.0-17bookworm) ...
root@a9f00190986d:/# apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8786 kB]
Get:5 http://apt.puppet.com bookworm InRelease [56.2 kB]
Get:6 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [13.8 kB]
Get:7 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [156 kB]
Get:8 http://apt.puppet.com bookworm/puppet7 all Packages [715 B]
Get:9 http://apt.puppet.com bookworm/puppet7 amd64 Packages [21.5 kB]
Fetched 9289 kB in 1s (11.1 MB/s)
Reading package lists... Done
root@a9f00190986d:/# apt install puppet-agent
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  puppet-agent
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 30.8 MB of archives.
After this operation, 109 MB of additional disk space will be used.
Get:1 http://apt.puppet.com bookworm/puppet7 amd64 puppet-agent amd64 7.30.0-1bookworm [30.8 MB]

I will check with release engineering about puppet-release-bookworm.deb

Hey @joshcooper, not sure re: old version of puppet installed, that is odd. but re: "adds bookworm support", yes that was added in #20, this PR was originally set up to implement those changes, but I have updated the title to reflect the new limited set of changes


Bug fix: incorrect font selected

Style is ignored when there is no font-style in registered font.
This bug can be reproduced at http://jsfiddle.net/tasior/z13Lgk1a/
Following condition is wrong:
(thefont.face["font-style"] == style || !thefont.face["font-style"])
When you register font without "font-style" and then another in same family and same weight it will pick the first one always.

Invalid code and no generated files. Please follow the contribution guidelines, thanks.


Float division by zero when fitting a model

Greetings!
I am replicating a paper, and need to run a simple OLS regression with a fixed effect. To do that I am running the PanelOLS function from linearmodels.  I'm estimating the effect of treatment on child mortality, here is the raw data:
AEJ2018_child_mortality_computation.zip
Here is what I do on the dataset:
data_2 = pd.read_stata("AEJ2018_child_mortality_computation.dta")
Collapsing to the sum:
data_2 = data_2.groupby(['villageid', 'branchid', 'treatment'], as_index = True)[['death_under5','count_month_u5', 'death_under1', 'count_month_u1','death_under1m','count_month_u1m']].sum().reset_index()
Generating variable of interest:
data_2['count_month_u5'] = data_2.apply(lambda row: row.count_month_u5/12, axis = 1)
data_2['mrate_u5'] = (data_2['death_under5']/data_2['count_month_u5'])*1000
Indexing:
data_2 = data_2.set_index(['villageid', 'branchid'], drop = False)
Model:
model = PanelOLS(data_2.mrate_u5, data_2.treatment, entity_effects = True, drop_absorbed=True) res = model.fit(cov_type = 'clustered', cluster_entity = True) print(res)
When fitting the model Python returns an error:
ZeroDivisionError                         Traceback (most recent call last)
 in
1 model = PanelOLS(data_2.mrate_u5, data_2.treatment, entity_effects = True, drop_absorbed=True)
----> 2 res = model.fit(cov_type = 'clustered', cluster_entity = True)
3 #print(res)
4 res
~\anaconda3\lib\site-packages\linearmodels\panel\model.py in fit(self, use_lsdv, use_lsmr, low_memory, cov_type, debiased, auto_df, count_effects, **cov_config)
1722             mu = 0
1723         total_ss = float((y - mu).T @ (y - mu))
-> 1724         r2 = 1 - resid_ss / total_ss
1725
1726         root_w = np.sqrt(self.weights.values2d)
ZeroDivisionError: float division by zero
At the same time, without fitting, i.e.:
model = PanelOLS(data_2.mrate_u5, data_2.treatment, entity_effects = True, drop_absorbed=True)    print(res)
everything works properly, and produces a valid regression output.
Equivalently, in R:
library(plm) model <- plm(mrate_u5~ treatment, data = df, index = c("branchid"), model = 'within') summary(model)
I'm okay without fitting, but I'm simply curious what I've done wrong this time. I've used PanelOLS for other regressions with fitting and it worked nicely. Thanks!
Thanks!
Edit: for authorship reasons, leaving a link where I got all the data, it was provided by authors of the paper I'm replicatin https://www.openicpsr.org/openicpsr/project/116355/version/V1/view
Edit2: after restarting the kernel, even without fitting doesn't work

Did you figure it out?

In your model, all of the "x" variables are absorbed by your entity effects. The model cannot be estimated.


Upgrade Issue templates

Resolves #6
Added Issue templates

@lohanidamodar Hi I have updated the templates

@Yashesvinee Great work. One more change before we can approve. We have made few changes in the language and also the format of Code of conduct. Please check https://github.com/appwrite/appwrite template and update accordingly.
Thank you.

@lohanidamodar I have made the required changes


Minor edits

One change fixes this strange rendering:

VSCode seems to expect something like this to happen, although I don't know markdown well enough to understand why.
Some changes to make things more consistent.

This strange rendering is so weird... Both the VSCode higheliger and @next/mdx (the markdown parser we are using) are understanding it as an element of an ordered list:
10250.

One workaround is to escape the .:
10250\.

What do you think?

I'm quite happy w/ using backticks, as seen in https://deploy-preview-1073--cert-manager-website.netlify.app/docs/troubleshooting/webhook/:

But, if you'd prefer to not decorate port numbers and instead use a backslash against the ., I can live with that. I'm not attached to any particular change. Anything that fixes the current rendering is an improvement.

I actually like the fact that these port numbers "stand out" thanks to the backticks. Let's go with that.
/lgtm


Use observations and move data loading upstream

See #495.

Re-triggered the build. Will merge once tests pass.


Support github.com/RoseSecurity/Kuzco

https://github.com/RoseSecurity/Kuzco

Thank you.

#27032

v4.222.0 is out 🎉
https://github.com/aquaproj/aqua-registry/releases/tag/v4.222.0


[BUG] Precision does not automatically update when zooming on coordinate map on a dashboard

What is the bug?
When a coordinate map visualization is used on a dashboard and 'Change precision on map zoom' is activated, it often happens that the precision level does not update correctly while zooming.
How can one reproduce the bug?
Steps to reproduce the behavior:

create index with mapping:
"mappings": { "properties": {  "location": {    "type": "geo_point"  } } }
index data to end up with ~100 documents with data such as this:
"_source" : { "some_data" : 92.47, "date" : "2022-03-10T09:33:28", "location" : "40.450655,11.510062" }
create a coordinate map visualization of the data and end up with something like this:
http://localhost:5601/app/visualize#/edit/7a0cad60-c148-11ec-b44a-e3ba16249f67?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(mapCenter:!(45.42737117898911,10.39583873351052),mapZoom:10),vis:(aggs:!((enabled:!t,id:'1',params:(field:some_data),schema:metric,type:max),(enabled:!t,id:'2',params:(autoPrecision:!t,field:location,isFilteredByCollar:!t,precision:5,useGeocentroid:!t),schema:segment,type:geohash_grid)),params:(addTooltip:!t,colorSchema:'Yellow%20to%20Red',heatClusterSize:1.5,isDesaturated:!t,legendPosition:bottomright,mapCenter:!(0,0),mapType:'Scaled%20Circle%20Markers',mapZoom:2,wms:(enabled:!f,options:(attribution:'',format:image%2Fpng,layers:'',styles:'',transparent:!t,version:''),selectedTmsLayer:(attribution:'%3Ca%20rel%3D%22noreferrer%20noopener%22%20href%3D%22https:%2F%2Fwww.openstreetmap.org%2Fcopyright%22%3EMap%20data%20%C2%A9%20OpenStreetMap%20contributors%3C%2Fa%3E',id:road_map,maxZoom:10,minZoom:0,origin:elastic_maps_service),url:'')),title:test_map,type:tile_map))
Add the visualization to a dashboard, save it, and start trying to zoom in and out. It looks like the precision is set to 2 while zooming out and stays fixed at 2, even when one zooms back in.

What is the expected behavior?
Precision level is updated according to zoom level.
What is your host/environment?

OS: Arch Linux
Opensearch Dashboards Version: 1.3.1
Plugins: none

Do you have any screenshots?
see https://discuss.opendistrocommunity.dev/t/map-visualizations-in-opensearch-dashboards/9000/9?u=adam
Do you have any additional context?
Probably the same issue as https://github.com/elastic/kibana/issues/74714

Hey guys, are there any news on this issue?

Hey @adamthor1 thanks for the follow up - this one had gone unseen in my backlog. Let me reproduce the scenario so that I can debug the issue. I'll keep this thread up to date with my findings.

I had some questions @adamthor1 -

Are you seeing this behavior when the following option is enabled?

Also, could you please send me sample urls for a couple of zoom configurations, wanted to verify few other things as well.

Hi @Shivamdhar
yes, the following option is enabled.
Here are some sample urls of the visualization. The bug is only visible when using a dashboard and not the visualization by itself.
http://localhost:5601/app/visualize#/edit/7a0cad60-c148-11ec-b44a-e3ba16249f67?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1y,to:now))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(mapCenter:!(45.42929873257377,10.404052734375002),mapZoom:10),vis:(aggs:!((enabled:!t,id:'1',params:(field:some_data),schema:metric,type:max),(enabled:!t,id:'2',params:(autoPrecision:!t,field:location,isFilteredByCollar:!t,precision:5,useGeocentroid:!t),schema:segment,type:geohash_grid)),params:(addTooltip:!t,colorSchema:'Yellow%20to%20Red',heatClusterSize:1.5,isDesaturated:!t,legendPosition:bottomright,mapCenter:!(0,0),mapType:'Scaled%20Circle%20Markers',mapZoom:2,wms:(enabled:!f,options:(attribution:'',format:image%2Fpng,layers:'',styles:'',transparent:!t,version:''),selectedTmsLayer:(attribution:'%3Ca%20rel%3D%22noreferrer%20noopener%22%20href%3D%22https:%2F%2Fwww.openstreetmap.org%2Fcopyright%22%3EMap%20data%20%C2%A9%20OpenStreetMap%20contributors%3C%2Fa%3E',id:road_map,maxZoom:10,minZoom:0,origin:elastic_maps_service),url:'')),title:test_map,type:tile_map))
http://localhost:5601/app/visualize#/edit/7a0cad60-c148-11ec-b44a-e3ba16249f67?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1y,to:now))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(mapCenter:!(45.42929873257377,10.404052734375002),mapZoom:9),vis:(aggs:!((enabled:!t,id:'1',params:(field:some_data),schema:metric,type:max),(enabled:!t,id:'2',params:(autoPrecision:!t,field:location,isFilteredByCollar:!t,precision:5,useGeocentroid:!t),schema:segment,type:geohash_grid)),params:(addTooltip:!t,colorSchema:'Yellow%20to%20Red',heatClusterSize:1.5,isDesaturated:!t,legendPosition:bottomright,mapCenter:!(0,0),mapType:'Scaled%20Circle%20Markers',mapZoom:2,wms:(enabled:!f,options:(attribution:'',format:image%2Fpng,layers:'',styles:'',transparent:!t,version:''),selectedTmsLayer:(attribution:'%3Ca%20rel%3D%22noreferrer%20noopener%22%20href%3D%22https:%2F%2Fwww.openstreetmap.org%2Fcopyright%22%3EMap%20data%20%C2%A9%20OpenStreetMap%20contributors%3C%2Fa%3E',id:road_map,maxZoom:10,minZoom:0,origin:elastic_maps_service),url:'')),title:test_map,type:tile_map))
http://localhost:5601/app/visualize#/edit/7a0cad60-c148-11ec-b44a-e3ba16249f67?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1y,to:now))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(mapCenter:!(45.42929873257377,10.404052734375002),mapZoom:8),vis:(aggs:!((enabled:!t,id:'1',params:(field:some_data),schema:metric,type:max),(enabled:!t,id:'2',params:(autoPrecision:!t,field:location,isFilteredByCollar:!t,precision:5,useGeocentroid:!t),schema:segment,type:geohash_grid)),params:(addTooltip:!t,colorSchema:'Yellow%20to%20Red',heatClusterSize:1.5,isDesaturated:!t,legendPosition:bottomright,mapCenter:!(0,0),mapType:'Scaled%20Circle%20Markers',mapZoom:2,wms:(enabled:!f,options:(attribution:'',format:image%2Fpng,layers:'',styles:'',transparent:!t,version:''),selectedTmsLayer:(attribution:'%3Ca%20rel%3D%22noreferrer%20noopener%22%20href%3D%22https:%2F%2Fwww.openstreetmap.org%2Fcopyright%22%3EMap%20data%20%C2%A9%20OpenStreetMap%20contributors%3C%2Fa%3E',id:road_map,maxZoom:10,minZoom:0,origin:elastic_maps_service),url:'')),title:test_map,type:tile_map))
While viewing it on a dashboard, the body of the request (POST  http://localhost:5601/internal/search/opensearch) is like this:
{"params":{"index":"python-test-index","body":{"aggs":{"filter_agg":{"filter":{"geo_bounding_box":{"ignore_unmapped":true,"location":{"top_left":{"lat":45.756750000000004,"lon":9.25323},"bottom_right":{"lat":45.09751,"lon":11.53839}}}},"aggs":{"2":{"geohash_grid":{"field":"location","precision":5},"aggs":{"1":{"max":{"field":"some_data"}},"3":{"geo_centroid":{"field":"location"}}}}}}},"size":0,"stored_fields":["*"],"script_fields":{},"docvalue_fields":[{"field":"date","format":"date_time"}],"_source":{"excludes":[]},"query":{"bool":{"must":[],"filter":[{"match_all":{}},{"range":{"date":{"gte":"2021-07-25T19:12:12.457Z","lte":"2022-07-25T19:12:12.457Z","format":"strict_date_optional_time"}}}],"should":[],"must_not":[]}}},"preference":1658776318457}}
When I zoom out once (which should be mapZoom:9), the precision changes to a value that does not fit:
{"params":{"index":"python-test-index","body":{"aggs":{"filter_agg":{"filter":{"geo_bounding_box":{"ignore_unmapped":true,"location":{"top_left":{"lat":46.085645,"lon":8.11065},"bottom_right":{"lat":44.767175,"lon":12.68097}}}},"aggs":{"2":{"geohash_grid":{"field":"location","precision":2},"aggs":{"1":{"max":{"field":"some_data"}},"3":{"geo_centroid":{"field":"location"}}}}}}},"size":0,"stored_fields":["*"],"script_fields":{},"docvalue_fields":[{"field":"date","format":"date_time"}],"_source":{"excludes":[]},"query":{"bool":{"must":[],"filter":[{"match_all":{}},{"range":{"date":{"gte":"2021-07-25T19:13:26.420Z","lte":"2022-07-25T19:13:26.420Z","format":"strict_date_optional_time"}}}],"should":[],"must_not":[]}}},"preference":1658776318457}}

Thanks for you response! I'll go through the links you have shared and post my findings here.

Hi @Shivamdhar did you find anything?

Hi @adamthor1 , we are developing new maps visualization plugin which will support different types of layers, the cluster layer(coordinate map) is targeting release at version 2.6.0, February 28th, 2023, so that it will unblock your use case.


modify doc about deploy of hadoop: deprecate KERB_TICKET_CACHE_PATH

I have create https://github.com/tensorflow/tensorflow/issues/12290 to declare this problem.

Can one of the admins verify this patch?

I signed it!

I signed it!


Improve toLowerCase and toUpperCase functions.

Language-sensitive mappings are not processed now.
Binary size: 136160	 -> 137684
"Fixes #323"

Please add "Fixes #323" to the commit message.

Where is the generator script?


Replace this with Bot.send_message

https://github.com/dadyarri/ralph/blob/10fd1fd40f91c3deae392c5dffa2ed09149bf61a/bot.py#L209

Removed in 445181ac50c45e15d429c84082684e233406deed.


Test and publish refactored put method

Lead: @kalluriramkumar
Is your feature request related to a problem? Please describe.
Test the refactored put method in at_client_impl and publish the packages.

[x] Merge changes to trunk branch in at_client
[x] Test the changes
[x] Publish the packages.

The changes are available in the latest at_client library: at_client: 3.0.10


ImageFolder improvements

This PR adds the following improvements to the imagefolder dataset:

skip the extract step for image files (as discussed in https://github.com/huggingface/datasets/pull/2830#discussion_r816817919)
option to drop labels by setting drop_labels=True (useful for image pretraining cc @NielsRogge). This is faster than loading a dataset and removing the label column because we don't need to iterate over the files to infer class labels.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Sorting vectors

Right now, it looks like we can only build a Vec by appending. It would be nice to be able to sort our vectors after they have been built. For example, I may want to compute a hilbert index for the nodes and sort them by that. That way, I can access the data with good locality.
Is there a straightforward way to sort?

Usually, what we sort data in memory and then write it to disk with e.g. ExternalVector. We had some experimental work on sorting ExternalVector externally, which is useful when the data does not fit on disk, but unfortunately we never finished it.

There are several problems with memory mapping mutable vectors:
A) It violates Rust's safety guarantees: A mutable reference invalidates all other references, but nothing stops another applications from opening the same file. We already violate this slightly by read-only memory mapping (somebody could manually open it writeable).
B) Just because it is mutably memory mapped doesn't mean you can sort it easily: Many sorting algorithms are not in-place, or optimized for external-memory sorting
C) Some flatdata structs are not self-contained, they also read data from the "next" item in the vector (@range attribution does that). Those types of structs must not be re-ordered.
That's why our general plan was to rather extend ExternalVector with a dedicated sorting method that is optimized for external memory, and can only be called when finalizing the vector (e.g. super-scalar-sample-sort, see https://github.com/lorenzhs/ssssort ), and prevents being used in case the struct is not self-contained.

The reason I ask is that I'm experimenting with creating tile indices using a Hilbert Curve. I am converting Node LatLons to their corresponding Hilbert number, then sorting Nodes by that. For ways, I do the same thing, with the Hilbert number derived from their Point on Surface.
osmflat-rs seems like a compelling fit for a back-end format, but an easy way of memmap sorting is needed.

Here's a general idea behind sorting vectors. Mutable accesors could be built into the storage code in flatdata to allow this sort of thing and maybe provide some guardrails...
https://github.com/boxdot/osmflat-rs/pull/74
It would be super cool if you could annotate the schema to make the sorting populate throughout all references from the sorted vector.
I haven't yet tried sorting the nodes themselves yet.

Building sorting into flatdata itself is definitely a feature we are looking forward to. Automatically updating any references is most likely not going to happen, since such a thing usually requires having a large in-memory index that remaps references, and also reordering data references with @range.
The nodes in osmflat itself are not sortable directly, though (as they have @range(tags) tag_first_idx: u64 : 40;), so one would most likely create a temporary intermediate storage for sorting.
Regarding the usability of osmflat as a backend format: While it can word for simple use cases I think it is much better suited as a input format to build your backend format: Most backends will need some of:

specialized indices (hilbert/zorder curves, quadtrees, tiles, etc)
pre-evaluated tags (convert tags into proper attributes, sanitize inputs, etc)
precomputed datastructures/algorithms (connect ways for routing, simplify polygons, assemble multi-polgyons, etc)

In my opinion osmflat is rather suited as an input format to build your own backend format out of: It is much easier to consume than PBF, requires less RAM (due to references being already resolved). We were mostly designing it as a data-exchange format replacing PBF.

Extending ExternalVector with a sort is a pretty interesting idea. Not sure if there is a parent issue on this.
Have you thought about radix sort?
https://docs.rs/radsort/latest/radsort/index.html

Radix sort is usually not the best choice if you need to sort data that does not fit into RAM: You would need one pass over the data for each 4/8/16 bits (whatever your word size is). E.g. assuming 16 bits words, and 32 byte structures you would need up to 16 passes of reading/writing the data. Some of the later passes might fit into memory, but it mostly depends on the data.
Compare that e.g. with (assuming that your data size is less than RAM ^ 1.5, e.g. 4 GB RAM -> less than 281 PB data ):

a super-scalar-sample-sort: It has one pass for classifying data, one pass for shuffling it around.
k-way merge sort: one pass to sort each block in RAM, one pass to merge all of them

Makes sense. Basically, the sort needs to be localized first to reduce paging.


Interface: Selected folder not coloured in accent color

Description

This Item could be selected or hovered over... it is selected!
When hovering over it it changes to the accent colour:

maybe the text should also change to white for darker accent colours while I'm at it, or use a different version of the colour for selection? just something to make it more readable
Steps To Reproduce
Go to any folder
Click on any file, but don't move your mouse
Move your mouse one pixel
Expected behavior
Accent colour for selected folders
Files Version
Version: <IP_ADDRESS> OS Version: 10.0.19043.1110
Windows Version
Windows 10 Home 21H1
Relevant Assets
No response

The listview selection syles only work correctly on Windows 11. Hopefully WinUI will fix it.


Modify theme to include time in minutes and list view

Using the modified version of this theme for my blog

Also added related posts at the bottom of the article


Update link to the ObjectGraph JavaDoc

Thanks! Haven't dug into Travis's woes, clearly nothing to do with this PR.