proposal: x/mobile: add permission request on android

What version of Go are you using (go version)?
go version go1.17.3 linux/amd64

Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
go env Output
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/matejmag305/.cache/go-build"
GOENV="/home/matejmag305/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/matejmag305/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/matejmag305/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/matejmag305/go_sdk/go1.17.3"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/matejmag305/go_sdk/go1.17.3/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build16419202=/tmp/go-build -gno-record-gcc-switches"

What did you do?
I was found problem: is no easy way how request permission by runtime..........
More story
I m interesting in programing android apps with go, I have created small library for my (experimental) project, the library is trying implement bluetooth socket ....., but I have problem with manifest (androids ignore Bluetooth permission........), I was trying to fix(by runtime), then I realize that there is not easy way how request permission by runtime.........., I was writing this problem to  fyne-io/fyne, which forms the graphics library of my experimental project, but now I think it is issue to gomobile.

Solution Proposal
The Solution would be use existing way how from ndk request permission. It would add two functions to programmer (user) interface. It is support with ndk solution:
More story
Add to code to mobile/app/android.c or somewhere where it pass best a few functions(or one, it depends on ordering of code) to background (in the fact I suggest the implementation solution, but it will be on implementation team how it will works) and two functions to user interface (the reason of this topic), pattern of these functions is the first answer in ndk request camera

Exported GO side (main Proposal)

 func RequestPermision(permission string)
Next code would use RunOnJVM from "mobileinit" package, it would have argument like "permission string" but in final version it could be "permission int", the code call `C.android_permissions(env, n)`, which could be void or return something, in further code is void ...
func RequestPermision(permission string){
	n := C.CString(permission)
	mobileinit.RunOnJVM(func(vm, jniEnv, ctx uintptr) error {
		env := (*C.JNIEnv)(unsafe.Pointer(jniEnv))
		C.android_permissions(env, n)// or result := ...
		return nil // or return result
	})
	C.free(unsafe.Pointer(n))
}

Other OS (IOS) should have: func RequestPermision(permission string){} // dummy function.
Of course probably IOS has similar system of permission, but I haven't experience for suggest any example...........

 func IsPermision(permission string) bool
The next exported function would have similar situation with argument like previous one - I do not know whether solution of type functions is the best..... and in the fact it: it would  unnecessary function, could be skipped because it could be execute at the beginning android_permissions, but sometime you want only find out whether permission is given and nothing more, therefore I mention this option .....
func IsPermision(permission string) bool{
	n := C.CString(permission)
        var result bool
	mobileinit.RunOnJVM(func(vm, jniEnv, ctx uintptr) error {
		env := (*C.JNIEnv)(unsafe.Pointer(jniEnv))
		result = bool(android_has_permission( env, perm_name))
		return nil // or return result
	})
	C.free(unsafe.Pointer(n))
        return result
}

Unexported - Background C side (of course only example how it could be done)

 void android_permissions(JNIEnv* env,  char* perm_name)
In this example it is void, but it could be string (char*) or bool and of course argument could be int, at the beginning function control whether permission would be granted and after that if it was not granted it would be call function to "real" request.:
void android_permissions(JNIEnv* env,  char* perm_name) {// or bool or string to indentify error
    bool OK = android_has_permission(
       env, perm_name
    ) ;
    if(!OK) {
       android_request_permission(env,perm_name ); // or return  android_request_file_permissions(env,perm_name ); due to error
    }
}

 void android_request_permission(JNIEnv* env, char* permissions)
It would be  call ndk java api to granted permission.
void android_request_permission(JNIEnv* env, char* permissions) { // or bool or string to reference error
    jobjectArray perm_array = (*env)->NewObjectArray(env,
      2,
      find_class(env, "java/lang/String"),
      (*env)->NewStringUTF(env, "")
    );
    (*env)->SetObjectArrayElement(env,
          perm_array, 1,
          android_permission_name(env, permissions)
        );
    (*env)->CallVoidMethod(
       current_class, request_permissions, perm_array, 0
    );
}

 bool android_has_permission(JNIEnv* env, const char* perm_name)
The function to check whether permission is granted:
bool android_has_permission(JNIEnv* env, const char* perm_name) {
    bool result = false;
    jstring ls_PERM = android_permission_name(env, perm_name);
    int PERMISSION_GRANTED = -1;
    {
       jclass ClassPackageManager = find_class(env, "android/content/pm/PackageManager" );
       jfieldID lid_PERMISSION_GRANTED = (*env)->GetStaticFieldID(env,
          ClassPackageManager, "PERMISSION_GRANTED", "I"
       );
       PERMISSION_GRANTED = (*env)->GetStaticIntField(env,
          ClassPackageManager, lid_PERMISSION_GRANTED
       );
    }
    {
       jclass ClassContext = find_class(env,  "android/content/Context"   );
       jmethodID MethodcheckSelfPermission = find_method(env, ClassContext,
       "checkSelfPermission", "(Ljava/lang/String;)I" );
       int int_result =(int) (*env)->CallIntMethod(env,
           current_class, MethodcheckSelfPermission, ls_PERM
       );
       result = (int_result == PERMISSION_GRANTED);
    }
    return result;
}

 jclass android_permission_name(JNIEnv* env, const char* perm_name)
The function to get permission class from (char*) string:
jclass android_permission_name(JNIEnv* env, const char* perm_name) {
    // nested class permission in class android.Manifest,
    // hence android 'slash' Manifest 'dollar' permission
    jclass ClassManifestpermission = find_class(env, "android/Manifest$permission");
    jfieldID lid_PERM = (*env)->GetStaticFieldID(env,
       ClassManifestpermission, perm_name, "Ljava/lang/String;"
    );
    jclass ls_PERM = (jclass)((*env)->GetStaticObjectField(env,
        ClassManifestpermission, lid_PERM
    ));
    return ls_PERM;
}

 static jmethodID request_permissions
And of course some functions could be "extracted" at init:
static jmethodID request_permissions;
//which init in function ANativeActivity_onCreate

Like show:  https://github.com/golang/mobile/blob/4e6c2922fdeed32d3596616518aaee7b0d79ce55/app/android.c#L71
and https://github.com/golang/mobile/blob/4e6c2922fdeed32d3596616518aaee7b0d79ce55/app/android.c#L78

I m sorry for my English, which is not best.......
EDITED:
Of course there is further reason, why implement this request: dangerous permission:  https://developer.android.com/training. Yes I know that golang isn't primary for android and majority of request will never need (because missing API), but  there are some permissions, which gomobile could use ..............., for example above mentioned Bluetooth.
ADDED LATER:
I have made some prototype of this (it is working on my android simulator): https://github.com/MatejMagat305/golang-prototype-permision (without adding to current code, only use function RunOnJVM) - it is only prototype which probably contains memory leak and maybe it should be good added or change functions to request multiple strings ............... And yes, something similar is on IOS, but I haven't suitable device, experience and internet claim that work a little bit diferent

How will the Go requestPermision function be called? Is it  to be called from  x/mobile itself, or user code? If this needs adding a public API, you may want to start a proposal (see https://golang.org/s/proposal ). Thanks.
cc @hyangah @eliasnaur @changkun

Thanks. So requestPermision will be an exported function?
As this adds a new API, following the proposal process ( https://golang.org/s/proposal ) may still be a good idea. I'll let x/mobile maintainers decide. Thanks.

yes function will be exported........., so now should I close this and open proposal isues without "Author background"?

I hope that it is finally in proposal form

above I mention jni option, but maybe it is not nessesery: https://developer.android.com/ndk/reference/group/permission

I have made some prototype of this (it is working on my android simulator): https://github.com/MatejMagat305/golang-prototype-permision

jni

To check a runtime permission, it seems NDK can be used, but with pid & uid?
To request one, according to [question] NDK request permissions, JNI is needed. So I think fyne needs to adopt this.

how can i import Java/java/nio/ByteBuffer with gobind?
when i use   gomobile.exe bind -target=android
it reponse cannot find package "Java/java/nio/ByteBuffer" in any of xxx
what class or function in android api can be import to go?

how can i import Java/java/nio/ByteBuffer with gobind? when i use gomobile.exe bind -target=android it reponse cannot find package "Java/java/nio/ByteBuffer" in any of xxx
what class or function in android api can be import to go?

you can not direcly do this, you must do this
// #include <jni>
// jobject some_function(...){ ... use jni ...}
// int value(...){ ... use jni ...>
import "C"

func main(){
    o := C.some_function(...) // call C jni
    v := int(C.value(o, ...))
}


Implement withdraw zones

Implement a withdraw area that you must extract to, instead of the 'dropship damage' approach. This makes it closer to the XCOM model, and seems to be what folks want in general.
CWolf has some functionality similar to this in MissionControl - see if that can be poached.

I didn't implement it. I mentioned it would be pretty easy to implement with the MC framework I'd created though. Was planning on it but then haven't had time these days
It would require a custom encounter logic set up
Basically an encounter logic object that is created that would detect if withdraw was allowed by any logic you want. Have the zone in but inactive on map load. Then detect when the user selects withdraw (probably an action fired off which is being watched for. Maybe on the message system as some things watch that in vanilla). Then when caught, do calculations on where the zone should go. Then move the inactive zone to that location and enable it. Mixture of chunks, actions and encounter logic and objectives
Sorry, pretty rough as I'd have to refresh myself again on the set up
Then you'd string that set up out in the MC encounter rule sets
Obviously you could do it without MC but it would be a lot of work, or very specific to that feature you were implementing
I have examples of set ups like that for the custom dialog in MC
More basic than the extraction points but it would still follow a similarish approach
@FrostRaptor


[BUG] Can not use localsave for layoutOrientation and iconSize

Environment
Self-Hosted (Docker)
System
Docker 20.10.3
Version
2.1.1
Describe the problem
When doing setting layoutOrientation and iconSize then save localy, I expect to see layoutOrientation and iconSize change, but I actually see everything stay same.
Check browser local storage, they have layoutOrientation and iconSize value same by my-local-conf.yml, not change anything.
Additional info
No response
Please tick the boxes

[X] You have explained the issue clearly, and included all relevant info
[X] You are using a supported version of Dashy
[X] You've checked that this issue hasn't already been raised
[X] You've checked the docs and troubleshooting guide
[X] You agree to the code of conduct

You could try a restart or rebuild through gui.

I think that there is a bug with iconSize at the moment, I've seen that too. Not with layoutOrientation though.
I'll take a look...


Cancel token

Hey there, congrats on being the 2nd most used 3rd party library in the Cocoa community.
I was starting to think about ways to work on cancellable image downloads within the Artsy iOS app. I want to be able to cancel networking for image thumbnails within a view controller on viewDidDisappear one of the methods that is looking like winning for me is tying our images together with a cancel token. This would mean creating an api like:
ARImageNetworkingToken *token =  [[ARImageNetworkingToken alloc] init];
[self.thumbnailImage ar_setImageWithURL:artwork.imageURL token:token];

With the aim that I can call [token pause] to kill all the networking requests when we go into another view controller. This also means having to create [token restart] when you get back into the viewController.
Given that all of this is a problem that every other app can potentially have, is there interest in me creating this at SDWebImage level instead? Given that it has access to shared networking state

I wonder if a simplification could be to have -[UIImageView sd_setImageWithURL:] and friends just return the internal SDWebImageOperation? This way the user can control it as they see fit, in our case that would be keeping an array of operations on the VC that can be cancelled when the view disappears.

@orta thanks. Cancelling is an issue we are currently looking at. There are 2 proposals, please take a look at #1069 and #883

Thanks @bpoplauschi - I bet I could build this on top of one of these #1069

@orta I'm actually looking at integrating #883, shouldn't take me more than a day or two to get that into master

Cool, that's even better ( I just figured that #1069 was closer to being merged )

@orta very soon, right after 3.7.4 gets out (probably today), I will merge #883.

No worries, I'm not trying to put pressure on you at all :+1: - you run one of the most popular libs, you can run it how you like

@orta Damn, I see now how long it took to get this merged. Finally, #883 is merged into our 4.x branch and we are full ahead towards a 4.x release. Do you still need this feature?

hah - yeah, it's not turn the ship around kind of thing, but it's I'll take this into account on upcoming artsy/eigen priorities  👍

👍

Guys, you can try the 4.0.0-beta version which is available via CocoaPods. An official release is on its way.

Congrats!


[Question]How can I upload a iso image by scp/http[s]?

My servers is hosted in a data center, and for security reason server hosted in this DC has no access to Internet. And ther is not a ftp server in this DC. And I can only connect to server's 22 & 30443 port by VPN. How can I upload  a iso file to harvester server?
My solution is: Use a physical server install a other OS (like Ubuntu) and config a ftp server. And use this internal ftp's url create a Ubuntu VM on harvester, and config this Ubuntu VM as a ftp server. And clear the previous physical Ubuntu ftp server, then install harvester on it. But it is too inconvenient to deploy a cluster in a new datacenter.

This is a v0.3.0 enhancement tracked in https://github.com/harvester/harvester/issues/570

Closing since there has been no recent activity and the linked support issue has been closed. Feel free to reopen if needed!


Fix vite build, its missing CSS

Not sure why vite is messing up all the CSS after prod build. Need to figure out how to fix it cause is blocking the release of the newer docker build for alpha 2

At the end wasn't a problem, the build was updating Tablr to version alpha 20 and this was breaking the CSS. Closing this task as no fix since no fix was done in the frontend, we just pinned the version to use alpha 17 release.


Implement request merging

When multiple clients access the same NAR file at the same time, we could download the file from the S3 bucket only once.

Add a concurrent lock on a per-derivation hash basis.
Associate the NAR stream to the lock.

type CachedBinaryCacheReader struct {
  Reader BinaryCacheReader
  Lock map[string]io.ReaderCloser
  Mutex sync.Mutex
}

func (r CachedBinaryCacheReader) GetFile(ctx context.Context, path string) (io.ReadCloser, error) {

}

Duplicate of #4?


[SPARK-45300][SQL][TESTS] Remove JDK 8 workaround in TimestampFormatterSuite

What changes were proposed in this pull request?
This PR removes the legacy workaround for JDK 8 in https://github.com/apache/spark/pull/28736.
Why are the changes needed?

We still need the main code for completeness, and in case there are other diff in the future JDK versions so this PR only fixes the tests.
We dropped JDK 11/8 at SPARK-44112

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Fixed unittests.
Was this patch authored or co-authored using generative AI tooling?
No.

cc @yaooqinn

I manually verified this. Merged to master.
$ build/sbt "catalyst/testOnly *.TimestampFormatterSuite"
...
[info] TimestampFormatterSuite:
[info] - explicitly forbidden datetime patterns (355 milliseconds)
[info] - SPARK-31939: Fix Parsing day of year when year field pattern is missing (50 milliseconds)
[info] - parsing timestamps using time zones (2 milliseconds)
[info] - format timestamps using time zones (6 milliseconds)
[info] - roundtrip micros -> timestamp -> micros using timezones (22 milliseconds)
[info] - roundtrip timestamp -> micros -> timestamp using timezones (8 milliseconds)
[info] - case insensitive parsing of am and pm (5 milliseconds)
[info] - format fraction of second (1 millisecond)
[info] - formatting negative years with default pattern (3 milliseconds)
[info] - parsing timestamp strings with various seconds fractions (8 milliseconds)
[info] - formatting timestamp strings up to microsecond precision (4 milliseconds)
[info] - SPARK-30958: parse timestamp with negative year (2 milliseconds)
[info] - SPARK-31557: rebasing in legacy formatters/parsers (10 milliseconds)
[info] - parsing hour with various patterns (3 milliseconds)
[info] - missing date fields (0 milliseconds)
[info] - missing year field with invalid date (1 millisecond)
[info] - missing am/pm field (1 millisecond)
[info] - missing time fields (1 millisecond)
[info] - missing hour field (0 milliseconds)
[info] - check result differences for datetime formatting (0 milliseconds)
[info] - SPARK-32424: avoid silent data change when timestamp overflows (1 millisecond)
[info] - SPARK-36418: default parsing w/o pattern (5 milliseconds)
[info] - SPARK-39193: support returning optional parse results in the default formatter (2 milliseconds)
[info] - SPARK-39280: support returning optional parse results in the iso8601 formatter (0 milliseconds)
[info] - SPARK-39281: support returning optional parse results in the legacy formatter (0 milliseconds)
[info] Run completed in 1 second, 193 milliseconds.
[info] Total number of tests run: 25
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 25, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 27 s, completed Sep 25, 2023, 8:14:13 PM


Remove code that adds "_T#" to sample IDs

Description of the bug
This is meant to force the sample IDs to be unique, but in our case we want to allow for duplicated rows with different group/reference combinations. Also, we dont want the cache to think that two smaples are different just because their sample IDs differ but their reads/reference do not.
Command used and terminal output
No response
Relevant files
No response
System information
No response

This is done


simplify some pattern matches to appease 1.68 clippy

The rust 1.68 clippy will start throwing errors about these that'd fail our ci.

bot merge


Any possiblity to remove mekanism recipe from crafting table?

Hi, dear Mekanism developers!
I've encountered a problem with recipe overriding for some mekanism items on my server.
In old MK version that well perfectly works with MekanismUtiles.removeRecipe(), but now it is'nt.
Maybe it's mechanics was changed? I will be very grateful for your help!

The mechanics haven't changed. Maybe try using a recipe remover tool?


Remove version: REALiX.HWiNFO version 7.02

all package installer URLs are broken
Microsoft Reviewers: Open in CodeFlow
part of #170837

URL: https://www.sac.sk/download/utildiag/hwi_702.exe
Response Error: The remote server returned an error: (404) Not Found.
(Automated message - build 920)


iBug: Blacklist [Nn]euro ?[Mm]ax

iBug requests the blacklist of the keyword [Nn]euro ?[Mm]ax. See the Metasmoke search here and the Stack Exchange search here.

[Nn]euro ?[Mm]ax has been seen in 0 true positives, 0 false positives, and 0 NAAs.


Allow gte, lte used on optional package

Previously this was always throwing an error when trying to compare a
missing dependency at build time. There is no error handling provided
anyway provided by this project, and detecting a missing dependency is
better done at addon's build entry.
close https://github.com/pzuraq/ember-compatibility-helpers/issues/38

CI is currently broken on master

Master is green now, @pzuraq @xg-wang did we still want to do this? Seems useful to be able to check versions of optional dependencies.

I think this is still useful. @pzuraq can we re-run the checks? Let me know if there's anything else to chnage

Can you rebase the PR? It appears to be running on Travis still, which is a mess, but the main branch has transitioned to GH actions

I've rebased this, but couldn't push to the original branch, so opened a PR https://github.com/pzuraq/ember-compatibility-helpers/pull/54

This also needed a test fix, now #54 is green

Thanks guys!


Current Atmel SAM flash driver will not erase last page

Describe the bug
I have been trying to erase the end of flash on an Atmel Sam4s(8b) using the current sam flash driver. When you attempt to erase the last page the driver returns an EFAULT error despite successfully erasing the last page. I believe this is because the success condition at https://github.com/zephyrproject-rtos/zephyr/blob/2f6b41a694ee47dac5c431a81b7673026b0908c3/drivers/flash/flash_sam.c#L397
cannot be met for the final page, as it requires a start_offset of the last page+1 to meet the condition. I am unsure of the best way to fix the issue, but currently in my application I have added the following check after the erase page function runs and it seems to be working:
/* Check if we've reached the end of pages to erase */
if (info->start_offset + info->size >= erase_data->section_end) {
         /* Succeeded, stop iterating */
         LOG_INF("Marking erase_data succeeded as true.");
         erase_data->succeeded = true;
         return false;
}

Note this can't replace the current check condition or else the final page would never be erased.
To Reproduce
Run flash_erase on an atmel sam chip for the last page of flash.
Expected behavior
No error is reported.
Impact
I believe I have solved the issue in my working tree, but I don't feel knowledgeable enough about the flash architecture to submit a PR.
Logs and console output
2024-03-17 16:52:44.852627:	[00:01:09.945,000] <dbg> flash_sam: sam_flash_erase: Erasing sector from 0x46000 to 0x80000
2024-03-17 16:52:44.863496:	[00:01:09.945,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x46000 of size 0x1000
2024-03-17 16:52:44.874021:	[00:01:09.965,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x47000 of size 0x1000
2024-03-17 16:52:44.879790:	[00:01:09.985,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x48000 of size 0x1000
2024-03-17 16:52:44.890265:	[00:01:10.005,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x49000 of size 0x1000
2024-03-17 16:52:44.900877:	[00:01:10.030,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x4a000 of size 0x1000
2024-03-17 16:52:44.906467:	[00:01:10.050,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x4b000 of size 0x1000
2024-03-17 16:52:44.917442:	[00:01:10.070,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x4c000 of size 0x1000
2024-03-17 16:52:44.927895:	[00:01:10.090,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x4d000 of size 0x1000
2024-03-17 16:52:44.938566:	[00:01:10.116,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x4e000 of size 0x1000
2024-03-17 16:52:44.943953:	[00:01:10.136,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x4f000 of size 0x1000
2024-03-17 16:52:44.954727:	[00:01:10.156,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x50000 of size 0x1000
2024-03-17 16:52:44.965497:	[00:01:10.176,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x51000 of size 0x1000
2024-03-17 16:52:44.971133:	[00:01:10.195,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x52000 of size 0x1000
2024-03-17 16:52:44.981563:	[00:01:10.215,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x53000 of size 0x1000
2024-03-17 16:52:44.992396:	[00:01:10.235,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x54000 of size 0x1000
2024-03-17 16:52:44.997922:	[00:01:10.255,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x55000 of size 0x1000
2024-03-17 16:52:45.008955:	[00:01:10.275,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x56000 of size 0x1000
2024-03-17 16:52:45.019503:	[00:01:10.295,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x57000 of size 0x1000
2024-03-17 16:52:45.029884:	[00:01:10.315,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x58000 of size 0x1000
2024-03-17 16:52:45.035813:	[00:01:10.341,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x59000 of size 0x1000
2024-03-17 16:52:45.046110:	[00:01:10.361,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x5a000 of size 0x1000
2024-03-17 16:52:45.056767:	[00:01:10.381,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x5b000 of size 0x1000
2024-03-17 16:52:45.062654:	[00:01:10.400,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x5c000 of size 0x1000
2024-03-17 16:52:45.073025:	[00:01:10.420,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x5d000 of size 0x1000
2024-03-17 16:52:45.083698:	[00:01:10.440,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x5e000 of size 0x1000
2024-03-17 16:52:45.094327:	[00:01:10.460,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x5f000 of size 0x1000
2024-03-17 16:52:45.099948:	[00:01:10.480,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x60000 of size 0x1000
2024-03-17 16:52:45.110736:	[00:01:10.500,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x61000 of size 0x1000
2024-03-17 16:52:45.121211:	[00:01:10.520,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x62000 of size 0x1000
2024-03-17 16:52:45.127038:	[00:01:10.540,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x63000 of size 0x1000
2024-03-17 16:52:45.137798:	[00:01:10.559,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x64000 of size 0x1000
2024-03-17 16:52:45.148319:	[00:01:10.585,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x65000 of size 0x1000
2024-03-17 16:52:45.158831:	[00:01:10.605,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x66000 of size 0x1000
2024-03-17 16:52:45.164528:	[00:01:10.625,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x67000 of size 0x1000
2024-03-17 16:52:45.175105:	[00:01:10.651,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x68000 of size 0x1000
2024-03-17 16:52:45.185737:	[00:01:10.671,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x69000 of size 0x1000
2024-03-17 16:52:45.192730:	[00:01:10.691,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x6a000 of size 0x1000
2024-03-17 16:52:45.201943:	[00:01:10.716,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x6b000 of size 0x1000
2024-03-17 16:52:45.212577:	[00:01:10.736,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x6c000 of size 0x1000
2024-03-17 16:52:45.223228:	[00:01:10.756,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x6d000 of size 0x1000
2024-03-17 16:52:45.229074:	[00:01:10.776,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x6e000 of size 0x1000
2024-03-17 16:52:45.239540:	[00:01:10.795,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x6f000 of size 0x1000
2024-03-17 16:52:45.250233:	[00:01:10.815,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x70000 of size 0x1000
2024-03-17 16:52:45.255787:	[00:01:10.835,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x71000 of size 0x1000
2024-03-17 16:52:45.266485:	[00:01:10.855,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x72000 of size 0x1000
2024-03-17 16:52:45.277139:	[00:01:10.875,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x73000 of size 0x1000
2024-03-17 16:52:45.282555:	[00:01:10.895,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x74000 of size 0x1000
2024-03-17 16:52:45.293350:	[00:01:10.921,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x75000 of size 0x1000
2024-03-17 16:52:45.304075:	[00:01:10.941,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x76000 of size 0x1000
2024-03-17 16:52:45.314846:	[00:01:10.960,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x77000 of size 0x1000
2024-03-17 16:52:45.320146:	[00:01:10.980,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x78000 of size 0x1000
2024-03-17 16:52:45.331103:	[00:01:11.000,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x79000 of size 0x1000
2024-03-17 16:52:45.341622:	[00:01:11.020,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x7a000 of size 0x1000
2024-03-17 16:52:45.347313:	[00:01:11.040,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x7b000 of size 0x1000
2024-03-17 16:52:45.357798:	[00:01:11.066,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x7c000 of size 0x1000
2024-03-17 16:52:45.368567:	[00:01:11.086,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x7d000 of size 0x1000
2024-03-17 16:52:45.379138:	[00:01:11.106,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x7e000 of size 0x1000
2024-03-17 16:52:45.384813:	[00:01:11.126,000] <dbg> flash_sam: sam_flash_erase_page: Erasing page at 0x7f000 of size 0x1000
2024-03-17 16:52:45.395485:	[00:01:11.145,000] <err> firmwareupdate: Failed to erase flash: -14. Cannot proceed with update

Environment (please complete the following information):

OS: MacOS
Zephyr SDK
Out of tree - but flash_sam.c and flash_page_layout.c (which invokes the function which produces the appropriate stop condition) are up to date with main.

Hi @adkinsjd ,
We really appreciate when community sends PR. Take a look at contribute-guidelines for more info about how to do it.

Hi @adkinsjd ,
Do you have plans to send the patch ?

Hi @bjarki-andreasen ,
Just checking. You made a lot of improvements on the SAM flash driver. I was wondering if you know about this issue.
Do you have any problems related with this?

I do see the bug in the code, I never ran into this since I did not use the last section of flash in my project, I will post a PR with a fix :)

@adkinsjd could you check out this PR and test it on your hardware? I no longer have hardware available to test with :)

or @nandojve could you test it?


Fix: use separate phases in buildPackageLock

Hi!
Recently, I encountered a case when I couldn't run just npm rebuild on all packages.
This PR:

splits the buildPhase in buildPackageLock into two phases
provides an example of overriding the buildPhase

Also, it fails when fetching wrappy (link)

Thanks! Happy to merge.

Nice! As I mentioned (link), there's a problem with wrappy that happens at preBuildPhase. I'll try to resolve it before merging.

Thank you!

Thank you for your cool library!


[Windows] Set PHP-7.4 as default

Description
Currently, during image generation process we install the latest version of PHP. At time of writing the current version of PHP is 8.0. One of the dependencies of PHP 8.0 is the visualstudio2019buildtools package which causes another issue  https://github.com/actions/virtual-environments/issues/1470.
2020-11-30T14:08:31.7820834Z     vhd: By installing you accept licenses for the packages.
2020-11-30T14:08:33.2822538Z     vhd: Progress: Downloading visualstudio2019buildtools <IP_ADDRESS>... 100%
2020-11-30T14:08:34.1117278Z     vhd: Progress: Downloading chocolatey-visualstudio.extension 1.8.1... 100%
2020-11-30T14:08:35.4864619Z     vhd: Progress: Downloading dotnetfx <IP_ADDRESS>90930... 100%
2020-11-30T14:08:37.0665171Z     vhd: Progress: Downloading chocolatey-dotnetfx.extension 1.0.1... 100%
2020-11-30T14:08:37.0669457Z     vhd: Progress: Downloading visualstudio-installer 2.0.1... 100%
2020-11-30T14:08:41.3145424Z     vhd: Progress: Downloading php 8.0.0... 100%

PS: choco list with the flag returns only the latest version of a package:
-a, --all, --allversions, --all-versions
     AllVersions - include results from all versions

Example:
PS > choco list -e php -a -r
php|8.0.0

Related issue:
https://github.com/actions/virtual-environments-internal/issues/1534
Check list

[x] Related issue / work item is attached
[ ] Tests are written (if applicable)
[ ] Documentation is updated (if applicable)
[ ] Changes are tested and related VM images are successfully generated

/azp run windows2016, windows2019

@gep13, Hi,
Do you have any idea how to parse all versions from https://chocolatey.org/packages/php#versionhistory list for PHP-7.4?
PS > choco list -e php -a -r
php|8.0.0

@al-cheb are you saying that this command:
choco list -e php -a -r

Isn't giving you a list of all versions?  I have just run it on my machine, and I get lots of versions returned:

@al-cheb actually... hold on a second...
The above screenshot was from the latest beta release of Chocolatey.  I can confirm that with 0.10.15 version of Chocolatey installed, I can replicate the issue that you are seeing.
If possible, can you upgrade to the latest pre-release of Chocolatey, i.e.
choco upgrade chocolatey --pre

And you will get all versions returned.

@gep13, Thank you. I will try to upgrade to the latest pre-version.

@al-cheb This should be fixed in chocolatey as vcredist140 should be enough to install PHP 8.0.0. It does not require visualstudio2019buildtools. I have created an issue there chocolatey-community/chocolatey-coreteampackages#1587

@shivammathur , thank you for logging an issue for choco package!
Yes, as Alex mentioned, installation of visualstudio2019buildtools on CI machines bring a lot of pain and break Visual Studio behavior.
On local box, you just have two separate command line tools for Visual Studio and Visual Studio Build Tools.
On CI machines, you can't select them this way and installation of Visual Studio Build Tools breaks some use-cases (override Visual Studio behavior).


[Bug]: shopify run dev for theme extensions local url incorrect.

Please confirm that you have:

[X] Searched existing issues to see if your issue is a duplicate. (If you’ve found a duplicate issue, feel free to add additional information in a comment on it.)
[X] Reproduced the issue in the latest CLI version.

In which of these areas are you experiencing a problem?
Extension
Expected behavior
When running npm run dev with a theme extension, the link to the local editor should be:
https://app-test-store-development.myshopify.com/admin/themes/148711473425/editor

Interacting with this link should open the local theme editor to enable and add the theme extension.
Actual behavior
The logs print out:
https://app-test-store-development.myshopify.com/admin/themes/148711473425/edit

Instead, the link fails and states the this is not available.
Verbose output
2023-05-05 05:33:18 │ extensions │ DEBUG POST https://partners.shopify.com/api/cli/graphql with request_id:
                                   7a6347e2-51fa-482a-b6d6-caa7e364ebe4
2023-05-05 05:33:18 │ extensions │ DEBUG {"data"=>{"extensionSpecifications"=>[{"name"=>"App Nav Item",
                                   "identifier"=>"app_nav_item", "options"=>{"managementExperience"=>"custom"},
                                   "features"=>{"argo"=>nil, {"name"=>"Online Store Nav Item",
                                   "identifier"=>"online_store_nav_item",
                                   "options"=>{"managementExperience"=>"custom"}, "features"=>{"argo"=>nil,
                                   {"name"=>"POS Link", "identifier"=>"pos_link",
                                   "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil,
                                   {"name"=>"App Link", "identifier"=>"app_link",
                                   "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Bulk Action", "identifier"=>"bulk_action",
                                   "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Flow - Action Definition", "identifier"=>"flow_action_definition",
                                   "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Flow - Trigger Definition", "identifier"=>"flow_trigger_definition",
                                   "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Flow - Trigger Discovery Webhook",
                                   "identifier"=>"flow_trigger_discovery_webhook",
                                   "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Privacy Compliance Webhooks",
                                   "identifier"=>"privacy_compliance_webhooks",
                                   "options"=>{"managementExperience"=>"custom"}, "features"=>{"argo"=>nil,
                                   {"name"=>"function", "identifier"=>"function",
                                   "options"=>{"managementExperience"=>"cli"}, "features"=>{"argo"=>nil,
                                   {"name"=>"App Link Runtime Parameters",
                                   "identifier"=>"app_link_runtime_params",
                                   "options"=>{"managementExperience"=>"custom"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Coinstall Retail Apps", "identifier"=>"coinstall_retail_apps",
                                   "options"=>{"managementExperience"=>"custom"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Produces Sales", "identifier"=>"produces_sales_extension",
                                   "options"=>{"managementExperience"=>"custom"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Checkout Post Purchase", "identifier"=>"checkout_post_purchase",
                                   "options"=>{"managementExperience"=>"cli"},
                                   "features"=>{"argo"=>{"surface"=>"checkout"}, {"name"=>"Checkout Extension",
                                   "identifier"=>"checkout_ui_extension",
                                   "options"=>{"managementExperience"=>"cli"},
                                   "features"=>{"argo"=>{"surface"=>"checkout"}, {"name"=>"Merchandising - Create
                                   Publication", "identifier"=>"merchandising_create_publication",
                                   "options"=>{"managementExperience"=>"custom"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Merchandising - Fulfillment Variant Details",
                                   "identifier"=>"merchandising_fulfillment_variant_details",
                                   "options"=>{"managementExperience"=>"custom"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Product Subscription", "identifier"=>"subscription_management",
                                   "options"=>{"managementExperience"=>"cli"},
                                   "features"=>{"argo"=>{"surface"=>"admin"}, {"name"=>"Marketing Activity",
                                   "identifier"=>"marketing_activity_extension",
                                   "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Web Pixel Extension", "identifier"=>"web_pixel_extension",
                                   "options"=>{"managementExperience"=>"cli"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Online Store - App Theme Extension",
                                   "identifier"=>"theme_app_extension",
                                   "options"=>{"managementExperience"=>"cli"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Payment Gateway", "identifier"=>"payment_gateway",
                                   "options"=>{"managementExperience"=>"custom"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Payments App - Redeemable ", "identifier"=>"payments_app_redeemable",
                                    "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Retail Cart Promotion", "identifier"=>"retail_cart_promotion",
                                   "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil,
                                   {"name"=>"Retail Product Recommendation",
                                   "identifier"=>"retail_product_recommendation",
                                   "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil,
                                   {"name"=>"POS UI Extension", "identifier"=>"pos_ui_extension",
                                   "options"=>{"managementExperience"=>"cli"},
                                   "features"=>{"argo"=>{"surface"=>"point_of_sale"}, {"name"=>"Subscription
                                   Link", "identifier"=>"subscription_link",
                                   "options"=>{"managementExperience"=>"dashboard"}, "features"=>{"argo"=>nil]
2023-05-05 05:33:18 │ extensions │ DEBUG GET https://app-test-store-development.myshopify.com/admin/api/unstable/t
                                   hemes/148711473425.json with request_id: c8678e76-23d2-465f-9f84-80dafd8a5772
2023-05-05 05:33:21 │ extensions │ DEBUG POST https://partners.shopify.com/api/cli/graphql with request_id:
                                   33a9aa45-f84d-4831-878c-627813a85626
2023-05-05 05:33:21 │ extensions │ DEBUG {"data"=>{"app"=>{"id"=>"40300150785", "title"=>"Pace (dev)",
                                   "apiKey"=>"5654af1232f1e0095ed3d283f897e785",
                                   "apiSecretKeys"=>[{"secret"=>"4746687f6c57fe4eb8d44a1472be5b17"}],
                                   "appType"=>"undecided", "extensionRegistrations"=>[{"id"=>"23910252545",
                                   "type"=>"THEME_APP_EXTENSION", "uuid"=>"f5179084-47e9-4878-a665-1d54a2167f07",
                                   "title"=>"pace-ui", "draftVersion"=>{"registrationId"=>"23910252545",
                                   "context"=>nil, "lastUserInteractionAt"=>"2023-05-05T01:22:32-04:00",
                                   "location"=>"https://partners.shopify.com/2588298/apps/40300150785/extensions/t
                                   heme_app_extension/23910252545", "validationErrors"=>[], "id"=>"93439590401",
                                   "uuid"=>"b33c7821-342a-4e64-bc5c-ff63fa91dfda", "versionTag"=>"0.0.0"]}
2023-05-05 05:33:21 │ extensions │
2023-05-05 05:33:21 │ extensions │ --------- Viewing extension…
2023-05-05 05:33:21 │ extensions │ Enable your theme app extension:
2023-05-05 05:33:21 │ extensions │ https://partners.shopify.com/2588298/apps/40300150785/extensions/theme_app_exte
                                   nsion/23910252545
2023-05-05 05:33:21 │ extensions │
2023-05-05 05:33:21 │ extensions │ Setup your theme app extension in the host theme:
2023-05-05 05:33:21 │ extensions │ https://app-test-store-development.myshopify.com/admin/themes/148711473425/edit
                                   or
2023-05-05 05:33:21 │ extensions │
2023-05-05 05:33:21 │ extensions │ Preview your theme app extension:
2023-05-05 05:33:21 │ extensions │ http://<IP_ADDRESS>:9292
2023-05-05 05:33:21 │ extensions │ ------------------
2023-05-05 05:33:21 │ extensions │ [2023-05-05 06:33:21] INFO  WEBrick 1.7.0
2023-05-05 05:33:21 │ extensions │ [2023-05-05 06:33:21] INFO  ruby 3.1.2 (2022-04-12) [x86_64-darwin20]
2023-05-05 05:33:21 │ extensions │ [2023-05-05 06:33:21] INFO  WEBrick::HTTPServer#start: pid=2509 port=9292
2023-05-05 05:33:22 │ extensions │ DEBUG POST https://partners.shopify.com/api/cli/graphql with request_id:
                                   f11c227a-e36e-45fe-981d-354c69bdfe61
2023-05-05 05:33:22 │ extensions │ DEBUG {"data"=>{"extensionUpdateDraft"=>{"extensionVersion"=>{"registrationId"=
                                   >"23910252545", "context"=>nil,
                                   "lastUserInteractionAt"=>"2023-05-05T01:33:22-04:00",
                                   "location"=>"https://partners.shopify.com/2588298/apps/40300150785/extensions/t
                                   heme_app_extension/23910252545", "validationErrors"=>[]}, "userErrors"=>nil}
2023-05-05 05:33:22 │ extensions │ Pushed » 'pace-test' to a draft
2023-05-05 05:33:22 │ extensions │ - blocks/star_rating.liquid
2023-05-05 05:33:22 │ extensions │ - snippets/stars.liquid
2023-05-05 05:33:22 │ extensions │ - locales/en.default.json
2023-05-05 05:33:22 │ extensions │ - assets/app.js
2023-05-05 05:33:22 │ extensions │ - assets/thumbs-up.png

Reproduction steps

Create a theme extension in the Shopify Node template.
npm run dev
In the terminal, click the link for the local theme editor.

Operating System
Mac OS Big Sur 11.7.4
Shopify CLI version (check your project's package.json if you're not sure)
3.45.0
Shell
iTerm2
Node version (run node -v if you're not sure)
v16.17.1
What language and version are you using in your application?
No response

Hi @liamjosephsilk!
Looking at your verbose output, seems like an issue with your terminal. I see this from your log:
2023-05-05 05:33:21 │ extensions │ https://app-test-store-development.myshopify.com/admin/themes/148711473425/edit
                                   or

Which means that the link is being split into a second line and your terminal breaks the link, some terminals can't interpret urls correctly if the are in two lines i'm afraid.
The solution would be to make your terminal wider or just copy paste the full link


Crash upon deleting emails

K9 mail crashes frequently, seemingly related to me deleting emails. The is sometimes a bit of delay though between deleting the emails and the crash. I've been able to reproduce this twice in the past few minutes, see the attached logs.
I'm running 5.738, Android 10, on a Nokia 7.2

A solution for #5408 should prevent the crash you're experiencing. But it wouldn't fix the underlying cause. The crash you're experiencing has been around for quite a while for a low percentage of users. But I have no idea what is causing it.

Good to know that I'm not the only one experiencing this bug. I'm sure this will get fixed some day. Keep up the good work!


AdminJS v7 - feedback, urgent issues, questions, discussions

This is a thread to aggregate all questions/feedback/urgent bugs/discussions related to the recently released version 7.
Feel free to share your ideas or let us know about any issues you're facing related to migration or usage.
What's new in version 7
 describes the largest changes in version 7.
Migration Guide will help you migrate your project to version 7.
Demo Application
Demo application also includes our new Themes feature. If you would like to see themes in action, use one of these credentials to sign in:

<EMAIL_ADDRESS>/ password - a dark theme
<EMAIL_ADDRESS>/ password - a theme which uses custom components to remove the sidebar and instead move resources to the top bar
<EMAIL_ADDRESS>/ password - an example of a custom theme which you can build yourself in your own project (source code)

If you notice that parts of our documentation are outdated or unclear, please share that in this thread as well.

Working on a migration of a super basic app with just 4-5 resources and mongoose schema setup. Did the pnpm / npm / yarn update and now the system can't find the packages anymore.
https://www.npmjs.com/package/@adminjs/mongoose the docs are wrong - still showing require statements.
import { Database as MongooseDatabase, Resource as MongooseResource } from '@adminjs/mongoose';

Using this like in the new example app is dropping the error I'm mentioning above.

Great work guys!
I'm really in favor of moving to ESM-only but I feel like one thing hasn't been thought through. It's cooperation with NestJS which still uses CJS and has no plans to move to ESM.
https://github.com/nestjs/nest/issues/7021
Do you have any solution for that?

@mszula until Nest.js moves to ESM or starts dual packaging, you will either have to set up AdminJS express/fastify way:
src/app.module.ts
import { Module } from '@nestjs/common';

import { AppController } from './app.controller';
import { AppService } from './app.service';

@Module({
  imports: [
    import('@adminjs/nestjs').then(({ AdminModule }) =>
      AdminModule.createAdminAsync({
        useFactory: () => ({
          adminJsOptions: { rootPath: '/' },
        }),
      }),
    ),
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

tsconfig.json
{
  "compilerOptions": {
    // ...
    "moduleResolution": "nodenext"
  }
}

Instead of import(...).then(...) in app.module.ts you can just set it up Express way in main.ts:
import { NestFactory, HttpAdapterHost } from '@nestjs/core';

import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  const AdminJS = await import('adminjs');
  const AdminJSExpress = await import('@adminjs/express');

  const expressApp = app.get(HttpAdapterHost).httpAdapter;
  const admin = new AdminJS.default({});

  expressApp.use(
    admin.options.rootPath,
    AdminJSExpress.default.buildRouter(admin),
  );

  await app.listen(3000);
}
bootstrap();

Note that these changes still rely on you not actually updating to ESM, so you will have to await import(<any adminjs package>) to have it work or just import CJS versions.
As for whether it is possible to still set it up Nest-way - maybe, I don't know. It should be in the interest of NestJS maintainers to proceed with that migration because we cannot impede our own product's development based on their choice of not supporting ESM. Technically ESM projects should be able to import CJS packages, but it doesn't seem to be the case here.

@ddresch Have you managed to fix your problem? From what I can see you just need to make changes to your tsconfig.json:
"moduleResolution": "nodenext",
"module": "nodenext",
"target": "esnext"

Yes – I fixed all my issues now. Thanks for your feedback!

TypeError: Unknown file extension ".json"

TypeError: Unknown file extension ".json"

import someJson from './path/to/some.json' assert { type: 'json' }

or use alternatives: https://www.stefanjudis.com/snippets/how-to-import-json-files-in-es-modules-node-js/

Can you show full error log? This could be Nest issue since json imports work fine with every other framework I tried.

TypeError: Unknown file extension ".json" for E:\application\admin-mode\node_modules\adminjs\lib\locale\de\translation.json

@Aladinndevolpment Looks like type assertions are supported from Node 17+

I am stuck with this issue after upgrading to 7 both themes and typeorm is not detecting though in node_modules it exist

@mickyarun This is the same problem @ddresch had, I think you just have to add:
"moduleResolution": "nodenext",
"module": "nodenext",
"target": "esnext"

to your tsconfig.json

@dziraf
is this how I will register the adapter for TypeORM on nestJS?

[BUG] iam new to Adminjs idk if this is worth reporting
iam using mongoose when i edit field with population this happend on dark theme its not visible
light theme

dark theme

How do I make a resource the parent instead of the database name on the sidebar?

[BUG] iam new to Adminjs idk if this is worth reporting iam using mongoose when i edit field with population this happend on dark theme its not visible
light theme
dark theme

Yes, this is a bug. Thank you for reporting, I passed it up.

@dziraf is this how I will register the adapter for TypeORM on nestJS?

The registerAdapter fragment should be somewhere else, even outside of bootstrap function.

After migration to v7, I always get an error "You have to implement action component for your Action" for custom resource action without component. Something was changed with that, or that's just a bug 😄 ?
test: {         actionType: 'resource',         component: false,         handler: async () => {           return {             notice: {               message: 'Zamówienia zostały pobrane.',               type: 'success',             },           };         },       },

After migration, I am not able to implement any custom page, keep getting no routes matched
Implemented same like demo application

Solve it by deleting .adminjs folder it was having old cached code

I am creating a bundle in CI/CD pipeline after upgrading to 7, I am getting the following error for the image , any help?

@mickyarun you need to add the componentLoader instance to the options object of the upload feature instance. Like stated in the error.

@ddresch I have added in already
import {componentLoader} from "../../../admin/components.js";
   uploadFeature({       componentLoader,       provider: {aws: {region: AWS_REGION, bucket: MERCHANT_OFFER_S3_BUCKET_NAME}},       validation: {mimeTypes: ["image/png", "image/jpg", "image/jpeg", "image/svg+xml", "image/webp"]},       uploadPath: (record, filename) =>integrations/${record.id()}/${filename},       properties: {         file: "integrationFile",         key: "integrationImage",         filePath: "integrationFilePath",         bucket: MERCHANT_OFFER_S3_BUCKET_NAME       }     })
My component file
`import {ComponentLoader} from "adminjs";
import path from "path";
import * as url from "url";
const __dirname = url.fileURLToPath(new URL(".", import.meta.url));
export const componentLoader = new ComponentLoader();
export const add = (url: string, componentName: string): string => componentLoader.add(componentName, path.join(__dirname, url));
export const CashbackQRCode = add("components/cashbackqrcode", "CashbackQRCode");`

@dziraf Any help on the above issue, its happening only in production with ADMIN_JS_SKIP_BUNDLE

@mickyarun  If you use @adminjs/bundler, did you set assetsCDN in AdminJS object?
@SOULPWRD is an entry file generated in .adminjs folder when you start the server?

@dziraf
I hope this message finds you well. I am writing to inquire about the progress on resolving the issue detailed in https://github.com/SoftwareBrothers/adminjs/issues/1347
While I understand that this bug may be deemed less significant, and potentially tedious to address, I was hoping that you could kindly provide me with an estimated timeframe for a resolution.
As a temporary workaround, we can turn Chrome's autocomplete feature off. However, it would be highly appreciated if an option to address this issue could be implemented as soon as possible.
If it is not possible for you to address this issue within a reasonable timeframe, please let me know if you would welcome a Pull Request from me to resolve the issue.
Thank you for your time and efforts on this matter.

[BUG] iam new to Adminjs idk if this is worth reporting iam using mongoose when i edit field with population this happend on dark theme its not visible
light theme
dark theme

Fixed in https://github.com/SoftwareBrothers/adminjs/pull/1495

Login Page stopped to show errors, e.g in case of incorrect credentials.
You can try to use wrong credentials in the demo app, it's reproducing.
https://demo.adminjs.co/admin/login

[BUG] idk if iam doing something wrong or if its specfic to version 7 or not
i am using mongooes adapter
i have a field name idAuthor Refrence to "user" collection
but when i name it idAuthor it doenst show me dropdown
and when i rename it to author it show dropdown

is there a way i can keep idAuthor with dropdown

assetsCDN

No I didn't added, all generated files are in .adminjs folder

[BUG] custom actions with component: false, still show a error page to implement component
Have taken the latest v7.0.6

The handler associated with the action is called but this page also pops up

Hi guys. I used Adminjs to build CMS for the express ts app and faced some issues. Could your guys help me to resolve them?
Issue 1:
I faced an issue with import libs. For example, I used @adminks/themes
import { dark, light, noSidebar } from '@adminjs/themes'

and I faced issues: Cannot find module @adminjs/themes or its corresponding type declarations.ts(2307)
Even though I installed it and it exists in the node_modules. also the same issue with @adminjs/design-system
Issue 2:
After I set up as an example in a document here: https://docs.adminjs.co/installation/plugins/express#simple
I faced an issue on the browser after the server returned admin page:

Uncaught Error: ThemeProvider: "theme" prop is required.
at D (hoist-non-react-statics.cjs.js:103:38)
at hoist-non-react-statics.cjs.js:103:38
at hoist-non-react-statics.cjs.js:103:38
at mountMemo (global.bundle.js:21484:20)
at Object.useMemo (global.bundle.js:21929:17)
at Object.useMemo (global.bundle.js:1682:23)
at Le (hoist-non-react-statics.cjs.js:103:38)
at renderWithHooks (global.bundle.js:20564:19)
at mountIndeterminateComponent (global.bundle.js:24328:14)
at beginWork (global.bundle.js:25841:17)

Hope for any suggestions from your guys.
Node: 18

"@AdminJS/express": "^6.0.0",
"@AdminJS/themes": "^1.0.0",
"adminjs": "^7.0.6""
"@AdminJS/design-system": "^4.0.2",

and tsconfig
{
  "compilerOptions": {
    "strict": true,
    "jsx": "react",
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "declaration": false,
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "allowSyntheticDefaultImports": true,
    "allowJs": true,
    "target": "ESNext",
    "sourceMap": false,
    "outDir": "./dist",
    "baseUrl": ".",
    "incremental": true,
    "skipLibCheck": true,
    "resolveJsonModule": true,
    "esModuleInterop": true
  },
  "include": ["./src", "src/index.ts", "./public"],
  "exclude": ["./dist", "node_modules"]
}

Hi,
something is wrong with the nested translations
i18next::translator: missingKey pl translation properties.isActive.true Yes
I have this key in my translations:

Hi,
something is wrong with the nested translations
i18next::translator: missingKey pl translation properties.isActive.true Yes
I have this key in my translations:

The column header in the table translates correctly, but the translation in the records is missing:

Hi I am facing issue with importing @adminjs/nestjs and @adminjs/objection packages I am getting error Cannot find module '@adminjs/objection' or its corresponding type declarations. and yes ive tried to put "moduleResolution": "nodenext",
"module": "nodenext",
"target": "esnext" in tsconfig and pacakges are in node_modules folder someone help me I am not doing anything else Ive just written import its throwing issue thats why no screenshots are passed

Hi,
how can I translate in sidebar navigation.name with avaliable translates in system

Hi,
something is wrong with the nested translations
i18next::translator: missingKey pl translation properties.isActive.true Yes
I have this key in my translations:

The column header in the table translates correctly, but the translation in the records is missing:

Having either the property translation or the nested properties (like *.true/*.false) works fine. It only breaks once you have both, and always seems to pick the resource ove the nested ones.

Hi,
something is wrong with the nested translations
i18next::translator: missingKey pl translation properties.isActive.true Yes
I have this key in my translations:

The column header in the table translates correctly, but the translation in the records is missing:

Having either the property translation or the nested properties (like *.true/*.false) works fine. It only breaks once you have both, and always seems to pick the resource ove the nested ones.

That's right, in an earlier version of AdminJs (6.*) this worked fine

Hi,
something is wrong with the nested translations
i18next::translator: missingKey pl translation properties.isActive.true Yes
I have this key in my translations:

The column header in the table translates correctly, but the translation in the records is missing:

Having either the property translation or the nested properties (like *.true/*.false) works fine. It only breaks once you have both, and always seems to pick the resource ove the nested ones.

That's right, in an earlier version of AdminJs (6.*) this worked fine

Seems to be because V06 just got the admin.locales and initialized the redux but V07 does a flatten+unflatten on the locales, so this *.true, *.false etc is now broken.

Hi, I wanted to add AdminJs to my expressjs and typescript project using  "adminjs": "^7.2.0" with "@adminjs/express": "^6.0.0", "@adminjs/sequelize": "^4.0.0",
and getting the error Cannot find module '@adminjs/sequelize' or its corresponding type declarations.
My index.ts
import express from 'express';
import AdminJs from 'adminjs';
import AdminJsExpress from '@adminjs/express';
import AdminJsSequelize from '@adminjs/sequelize';
import * as allModels from './models';

const host = process.env.HOST ?? 'localhost';
const port = process.env.PORT ? Number(process.env.PORT) : 3000;
const app = express();

AdminJs.registerAdapter({
  Resource: AdminJsSequelize.Resource,
  Database: AdminJsSequelize.Database,
});

const localModel = Object.values(allModels);
localModel.pop(); // Remove Sequelize instance
localModel.pop(); // Remove Sequelize instance

const adminOptions =
  {
    // We pass Category to `resources`
    resources: localModel,
  }

const admin = new AdminJs(adminOptions)
const adminRouter = AdminJsExpress.buildRouter(admin)

app.use(admin.options.rootPath, adminRouter);

app.get('/', (req, res) => {
  res.send({ message: 'Hello API' });
});

app.listen(port, host, () => {
  console.log(`[ ready ] http://${host}:${port}`);
});

My tsconfig.base.json
{
  "compileOnSave": false,
  "compilerOptions": {
    "rootDir": ".",
    "sourceMap": true,
    "declaration": false,
    "moduleResolution": "node",
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "importHelpers": true,
    "target": "es2015",
    "module": "esnext",
    "lib": ["es2020", "dom"],
    "skipLibCheck": true,
    "skipDefaultLibCheck": true,
    "baseUrl": ".",
    "paths": {}
  },
  "exclude": ["node_modules", "tmp"]
}

Besides that I tried to use AdminJs version 6 but when NPM tries to resolve the version of @adminjs/express and @admin/sequelize it can not resolve the versions of the dependencies in a straigthforward way.
Any advice how is the best way to proceed?
Thanks in advance
Juan Manuel

Hi, I wanted to add AdminJs to my expressjs and typescript project using  "adminjs": "^7.2.0" with "@adminjs/express": "^6.0.0", "@adminjs/sequelize": "^4.0.0",
and getting the error Cannot find module '@adminjs/sequelize' or its corresponding type declarations.
My index.ts
import express from 'express';
import AdminJs from 'adminjs';
import AdminJsExpress from '@adminjs/express';
import AdminJsSequelize from '@adminjs/sequelize';
import * as allModels from './models';

const host = process.env.HOST ?? 'localhost';
const port = process.env.PORT ? Number(process.env.PORT) : 3000;
const app = express();

AdminJs.registerAdapter({
  Resource: AdminJsSequelize.Resource,
  Database: AdminJsSequelize.Database,
});

const localModel = Object.values(allModels);
localModel.pop(); // Remove Sequelize instance
localModel.pop(); // Remove Sequelize instance

const adminOptions =
  {
    // We pass Category to `resources`
    resources: localModel,
  }

const admin = new AdminJs(adminOptions)
const adminRouter = AdminJsExpress.buildRouter(admin)

app.use(admin.options.rootPath, adminRouter);

app.get('/', (req, res) => {
  res.send({ message: 'Hello API' });
});

app.listen(port, host, () => {
  console.log(`[ ready ] http://${host}:${port}`);
});

My tsconfig.base.json
{
  "compileOnSave": false,
  "compilerOptions": {
    "rootDir": ".",
    "sourceMap": true,
    "declaration": false,
    "moduleResolution": "node",
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "importHelpers": true,
    "target": "es2015",
    "module": "esnext",
    "lib": ["es2020", "dom"],
    "skipLibCheck": true,
    "skipDefaultLibCheck": true,
    "baseUrl": ".",
    "paths": {}
  },
  "exclude": ["node_modules", "tmp"]
}

Besides that I tried to use AdminJs version 6 but when NPM tries to resolve the version of @adminjs/express and @admin/sequelize it can not resolve the versions of the dependencies in a straigthforward way.
Any advice how is the best way to proceed?
Thanks in advance
Juan Manuel

I had that issue as well first, but setting module and moduleResolution to "NodeNext" in the tsconfig.json seems to resolve that issue

Like @Aladinndevolpment I'm in a catch-22 with the JSON import.
Using express with javascript.
Using nodejs 17.3 I get this when I try to start app.js (this is the app.js from the getting started documentation)
node:internal/errors:464
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".json" for /Users/rick/projects/rasi/taxservice/node_modules/adminjs/lib/locale/de/translation.json
    at new NodeError (node:internal/errors:371:5)
    at Object.file: (node:internal/modules/esm/get_format:72:15)
    at defaultGetFormat (node:internal/modules/esm/get_format:85:38)
    at defaultLoad (node:internal/modules/esm/load:22:14)
    at ESMLoader.load (node:internal/modules/esm/loader:359:26)
    at ESMLoader.moduleProvider (node:internal/modules/esm/loader:280:58)
    at new ModuleJob (node:internal/modules/esm/module_job:66:26)
    at ESMLoader.#createModuleJob (node:internal/modules/esm/loader:297:17)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:261:34)
    at async ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:81:21) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

Node.js v17.3.0

If I try to go back to an earlier nodejs (e.g. 16.13) it fails on the assert (which requires 17+).
I'm two pages into the getting started tutorial and completely blocked.

Calls to create certain models get stuck (the network request status is 'pending' forever) always, they don't even access the before hooks, any idea why?

For those integrating AdminJS with a NestJS monorepo, follow the steps below to configure it properly.

Modify the tsconfig.app.json File:

{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "moduleResolution": "nodenext",
    "module": "nodenext",
    "target": "esnext",
    "declaration": false,
    "outDir": "../../dist/apps/admin"
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules", "dist", "test", "**/*spec.ts"]
}

Disable webpack option in nest-cli.json

Set webpack to false inside compilerOptions
Example, my sub-app named admin
"admin": {
      "type": "application",
      "root": "apps/admin",
      "entryFile": "main",
      "sourceRoot": "apps/admin/src",
      "compilerOptions": {
        "webpack": false,
        "tsConfigPath": "apps/admin/tsconfig.app.json"
      }
    }

@mickyarun If you use @adminjs/bundler, did you set assetsCDN in AdminJS object?
@SOULPWRD is an entry file generated in .adminjs folder when you start the server?

Hi, yes it is.
import Component0 from '../node_modules/@adminjs/passwords/components/edit'
AdminJS.UserComponents.Component0 = Component0
import JsonbFieldShow from '../src/JsonbShow'
AdminJS.UserComponents.JsonbFieldShow = JsonbFieldShow
import JsonbFieldEdit from '../src/JsonbEdit'
AdminJS.UserComponents.JsonbFieldEdit = JsonbFieldEdit

Sorry for a late reply. I was waiting until the version 7 gets a bit stable. Unfortunately, the problem still persists.

Hi @dziraf, I'm facing issues with deleting rows with Hapi and TypeORM. Just created a new project and tried again.

Hi,
something is wrong with the nested translations
i18next::translator: missingKey pl translation properties.isActive.true Yes
I have this key in my translations:

The column header in the table translates correctly, but the translation in the records is missing:

Having either the property translation or the nested properties (like *.true/*.false) works fine. It only breaks once you have both, and always seems to pick the resource ove the nested ones.

That's right, in an earlier version of AdminJs (6.*) this worked fine

Seems to be because V06 just got the admin.locales and initialized the redux but V07 does a flatten+unflatten on the locales, so this *.true, *.false etc is now broken.

Hey @dziraf do you have in mind when you will be able to fix this issue?

Hi, I see adminjs/mongoose is using MongooseModel.count, but this function is deprecated. I got error. Please check and update.

Same here! MongooseModel.count() is deprecated! Will it be fixed in the next version?

Hey guys, a minor issue I found while adding filters on page navigation.
The following to options act differently.
options: {
href: () => undefined,
href: undefined
}
The first  one hides the resource from the navigation, but it is still acccessible via URL, while the second correctly shows the resource in navigation.

Need help please.
I want to create a mono repo with 3 folders:

adminjs
api
shared

Adminjs, created with adminjs create (v7 on nest.js). As written in documents, it only supports ESM.
The api is a simple nest application with mongoose. As I understand it, it only supports CJS.
A shared folder is a folder that contains some nestjs/mongoose utilities and schemas with decorators.
Maybe someone has faced a similar problem. How to describe package.json and tsconfig to connect a shared folder to each application together. Maybe someone can describe an example please. Or maybe someone sees some available repositories to look at?
I have already tried to make shared folders for build/esm/... and build/cjs/... But it doesn't help and I have problems after building the main applications. Each time there are different import issues and I can't solve them.
I am grateful to everyone who responds.

Cannot override Login page in v7. Tried adding componentLoader.override() in various files (app, config options, index) to no avail. Any suggestions? Thanks

Is there a simple way to use buildAuthenticatedRouter (express) using 'username' instead of 'email' as the required parameter? I noticed it's coded in types.ts but it's difficult to replace 'authenticate' (AuthenticationOptions) in practice (app.ts). The reason is that in my application a person can reuse its email address (e.g. as staff with multiple roles, as a tenant with multiple shops, or a customer doing business with different tenants) but is always constrained to a distinct username for each instance. So far AdminJS is great, thanks!


Typing the word test causes a black box to paint over content

The screen shot shows the black box (Right below src.test.html).  It also locks up the cursor and does not allow it to move forward.  If I move backwards the black box disappears.

@oleersoy Which version of Atom are you using and can you reproduce this in safe mode?

@izuzak I'm using 1.0.7 and it does not happen in safe mode.

@oleersoy Sounds like a problem related to one of the packages you have installed. Recommendation: try disabling packages one-by-one to figure out which one it is an report the problem on that package's issue tracker.

OK - Will do.  Thanks.


analyzer: Correct balance of special addresses

SUMMARY

Maintain the correct balance of special reserved accounts/addresses such as oasis1qrmufhkkyyf79s5za2r8yga9gnk4t446dcy3a5zm.
ISSUE TYPE

Feature Idea

COMPONENT NAME

Analyzer
ADDITIONAL INFORMATION

Relates to #123
https://app.clickup.com/t/863frdmb0

Resolved by #188


Generate cert also supports SubjectFormat option.

LGTM. Please return an error in case of absent host, then we can submit this one.


Logseq Markmap discrepancy

Hi,
For some reason, the miA theme is not compatible with the Logseq Markmap (v0.2.1) plugin.
The heading container (?) of a page seems to cover the plugin.

This was a personal preference that was left in when I released the theme publicly. I've fixed this in v1.1.4.

appreciated, thanks!


Trouble getting started with local files

Hi! I am going through the quick start to try out kerchunk (finally!). Unfortunately I hit what is to me a weird error and I am not sure what direction to go, being new to the library and the error.
I'm following the steps from the Quick Start: https://fsspec.github.io/kerchunk/test_example.html#
I am using local netcdf files, not from S3. Perhaps this is the issue because all the examples I can find are for files on S3 instead of local files.
Here is my code, though it isn't reproducible since the files are local
import kerchunk.hdf
import fsspec
import pathlib
from kerchunk.combine import MultiZarrToZarr

base = pathlib.Path("/mnt/depot/data/packrat/prod/aoos/nwgoa/processed/")
urls = sorted(base.glob("1999/*1999-01*.nc"))

so = dict(
    # anon=True,
    default_fill_cache=False, default_cache_type='first'
)
singles = []
for u in urls:
    with fsspec.open(u, **so) as inf:
        h5chunks = kerchunk.hdf.SingleHdf5ToZarr(inf, u, inline_threshold=100)
        singles.append(h5chunks.translate())

# Multi-file JSONs
mzz = MultiZarrToZarr(
    singles,
    # remote_protocol="s3",
    remote_options={'anon': True},
    concat_dims=["ocean_time"],
)

out = mzz.translate()

The error is  ValueError: Shuffle buffer is not an integer multiple of elementsize and here is the stack trace

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[30], line 9
      1 mzz = MultiZarrToZarr(
      2     singles,
      3     remote_protocol='file',
   (...)
      6     concat_dims=["ocean_time"],
      7 )
----> 9 out = mzz.translate()
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/kerchunk/combine.py:496, in MultiZarrToZarr.translate(self, filename, storage_options)
490 """Perform all stages and return the resultant references dict
491
492 If filename and storage options are given, the output is written to this
493 file using ujson and fsspec instead of being returned.
494 """
495 if 1 not in self.done:
--> 496     self.first_pass()
497 if 2 not in self.done:
498     self.store_coords()
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/kerchunk/combine.py:258, in MultiZarrToZarr.first_pass(self)
256 z = zarr.open_group(fs.get_mapper(""))
257 for var in self.concat_dims:
--> 258     value = self._get_value(i, z, var, fn=self._paths[i])
259     if isinstance(value, np.ndarray):
260         value = value.ravel()
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/kerchunk/combine.py:226, in MultiZarrToZarr._get_value(self, index, z, var, fn)
224     o = z[var].attrs[item]
225 elif selector.startswith("data:"):
--> 226     o = z[selector.split(":", 1)[1]][...]
227 elif selector.startswith("cf:"):
228     import cftime
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/zarr/core.py:844, in Array.getitem(self, selection)
842     result = self.get_orthogonal_selection(pure_selection, fields=fields)
843 else:
--> 844     result = self.get_basic_selection(pure_selection, fields=fields)
845 return result
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/zarr/core.py:970, in Array.get_basic_selection(self, selection, out, fields)
968     return self._get_basic_selection_zd(selection=selection, out=out, fields=fields)
969 else:
--> 970     return self._get_basic_selection_nd(selection=selection, out=out, fields=fields)
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/zarr/core.py:1012, in Array._get_basic_selection_nd(self, selection, out, fields)
1006 def _get_basic_selection_nd(self, selection, out=None, fields=None):
1007     # implementation of basic selection for array with at least one dimension
1008
1009     # setup indexer
1010     indexer = BasicIndexer(selection, self)
-> 1012     return self._get_selection(indexer=indexer, out=out, fields=fields)
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/zarr/core.py:1388, in Array._get_selection(self, indexer, out, fields)
1385 if math.prod(out_shape) > 0:
1386     # allow storage to get multiple items at once
1387     lchunk_coords, lchunk_selection, lout_selection = zip(*indexer)
-> 1388     self._chunk_getitems(
1389         lchunk_coords,
1390         lchunk_selection,
1391         out,
1392         lout_selection,
1393         drop_axes=indexer.drop_axes,
1394         fields=fields,
1395     )
1396 if out.shape:
1397     return out
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/zarr/core.py:2228, in Array._chunk_getitems(self, lchunk_coords, lchunk_selection, out, lout_selection, drop_axes, fields)
2226 for ckey, chunk_select, out_select in zip(ckeys, lchunk_selection, lout_selection):
2227     if ckey in cdatas:
-> 2228         self._process_chunk(
2229             out,
2230             cdatas[ckey],
2231             chunk_select,
2232             drop_axes,
2233             out_is_ndarray,
2234             fields,
2235             out_select,
2236             partial_read_decode=partial_read_decode,
2237         )
2238     else:
2239         # check exception type
2240         if self._fill_value is not None:
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/zarr/core.py:2141, in Array._process_chunk(self, out, cdata, chunk_selection, drop_axes, out_is_ndarray, fields, out_selection, partial_read_decode)
2139 except ArrayIndexError:
2140     cdata = cdata.read_full()
-> 2141 chunk = self._decode_chunk(cdata)
2143 # select data from chunk
2144 if fields:
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/zarr/core.py:2409, in Array._decode_chunk(self, cdata, start, nitems, expected_shape)
2407 if self._filters:
2408     for f in reversed(self._filters):
-> 2409         chunk = f.decode(chunk)
2411 # view as numpy array with correct dtype
2412 chunk = ensure_ndarray_like(chunk)
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/numcodecs/shuffle.py:50, in Shuffle.decode(self, buf, out)
49 def decode(self, buf, out=None):
---> 50     buf, out = self._prepare_arrays(buf, out)
52     if self.elementsize <= 1:
53         return out  # no shuffling needed
File ~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/numcodecs/shuffle.py:35, in Shuffle._prepare_arrays(self, buf, out)
32     return buf, out
34 if buf.nbytes % self.elementsize != 0:
---> 35     raise ValueError("Shuffle buffer is not an integer multiple of elementsize")
37 return buf, out
ValueError: Shuffle buffer is not an integer multiple of elementsize

Thanks for any ideas!

This may help: https://ncar.github.io/esds/posts/2023/kerchunk-mom6/

Should I be using the NetCDF3ToZarr backend for netcdf files, instead of the hdf backend?

Should I be using the NetCDF3ToZarr backend for netcdf files, instead of the hdf backend?

You need to use the appropriate one! The first few bytes will be b"CDF" for netCDF3 and b"HDF" for netCDF4/hdf backend if memory serves.

I think that was a red herring — they are netcdf4 files.
I just tried local files from another model and they immediately work, as in the steps I posted in run without error. So, I guess the problem is a quirk of the model files I used.
Still, I would ultimately like to use kerchunk on the original model files so if anyone has any ideas where to go in terms of debugging for that error, I would appreciate hearing them.

The error from shuffle doesn't mean much to me. It may be the case that numcodecs' shuffle implementation (numcodecs._shuffle._foUnshuffle) is more limited than hdf's, but I don't know why that might be. It is a pretty short function.

Grid information is not included in the model output files that aren't working, including lon/lat coordinates and many other grid variables. Could that be relevant? I mention it because it I am working with both of these models side by side and I frequently have to account for that as you can imagine. Not sure what else is obviously different between the two.

Sorry I missed this yesterday.
Chunks aren't specified in the individual files, if that is what you mean. I retried this with just 2 model files to narrow the scope and got the same error.

Chunks aren't specified in the individual files

Sorry, I don't follow. Can you show the data structure of the two files separately? Could it be that one has shuffle but the other no?

I think you want ncdump -sh  . It'll tell you about _ChunkSizes and shuffle things too i think

Oh ok, here those are:
ncdump1.txt
ncdump2.txt

It seems like actually the individual files don't load (i.e., you can't access their data - but maybe only one of the variables), before combination?
The two files look identical to each other, so perhaps there is a difference between these and another that works OK.
Perhaps next I would make run pdb following the exception and go up as far as "~/miniconda3/envs/xp-inventory/lib/python3.11/site-packages/kerchunk/combine.py:226" in the stack to see the state of the particular zarr data that is being accessed.

Ok good point. I dove into the code there and get:
z[selector.split(":", 1)[1]] returns <zarr.core.Array '/ocean_time' (24,) float64>
but
z[selector.split(":", 1)[1]][...] returns the error *** ValueError: Shuffle buffer is not an integer multiple of elementsize
I am not familiar with [...] and can't seem to google it. What does that do at the end of that line?

Ah I see as I am working in my other issue #343 that the [...] pulls out the numbers, maybe like [:] when working with netcdf datasets.
Maybe I need to go farther back in the code to see why those dates/values can't be accessed then.

I had a pretty random guess to try to figure out why one model was working and the other wasn't and compare the ocean_time variable attributes. I changed the calendar using ncap2 from nco from "gregorian" to "proleptic_gregorian" in the model that wasn't working and now it works.
ncap2 -O -s 'ocean_time@calendar="proleptic_gregorian"' nwgoa_1999-02-01.nc nwgoa_out.nc

The issue ended up being from reading in with zarr from what I can tell.

What I said in the last comment is wrong.
It appears that doing anything with ncap2 to the file fixes this issue. Using the original file, locally, leads to the shuffle error, but using that same file run through ncap2 with some arbitrary (as far as I can tell) change to an attribute (including on a different variable than ocean_time) change does not have the shuffle error.
I don't see any difference in the ncdump -hs between an unaltered and altered file ocean_time variable (when I don't modify ocean_time), but again the modified file moves past the error brought up on ocean_time.
Does ncap2 perform some basic change to the file that could account for this?

Does ncap2 perform some basic change

I have absolutely no idea what ncap2 might be doing. If you look through the references kerchunk generated for the two versions of the file, perhaps you can see a difference there for the appropriate variable. You can just look at the references dict directly (even with test-search in any editor), or use a combination of .ls and _cat_common on the filesystem to find the details of given references.

the [...] pulls out the numbers, maybe like [:]

... (builtins.Ellipsis) means "the whole of any number of axes", but : (builtins.slice(None)) means "the whole of one axis", so for a 1d array they are equivalent, but for 2d [...]->[:, :].  This is a numpy convention: builtins.slice(None)

Just a few updates:

simply running ncks <infile> <outfile> and using  with kerchunk fixes this problem, though the files are "identical"
the original files do indeed open fine with netcdf4 and xarray, but I guess not with zarr/kerchunk (at the zarr part of the kerchunk code)
with @lukecampbell's help it seems like the error might be related to compression I used when I processed the files originally, but still really hard to pin down or even see differences in the original and ncks-produced files.

I am going to rewrite the files with ncks and use those, so I will close this issue.

Ok I have an actual answer now, though the solution is the same as I said before, or you can avoid the problem in the first place.
This information and steps below come from @ajelenak who helpfully corresponded by email to figure it out though a chance connection through @hyoklee.
The error message "Shuffle buffer is not an integer multiple of elementsize" very likely means that the Shuffle codec used by the kerchunk package expects that the (chunk) buffer's size to process is exactly the product of chunk’s number of array elements and the byte size of the element’s datatype.
The error was caught on the ocean_time variable when I dug into the kerchunk code so I ran h5dump -d ocean_time -p [netcdf_file_causing_error.nc] and it showed
   FILTERS {
      PREPROCESSING SHUFFLE
      COMPRESSION DEFLATE { LEVEL 8 }
      COMPRESSION DEFLATE { LEVEL 3 }
   }

We also looked at the kerchunk output for "ocean_time/.zarray" which showed
'{"chunks":[512],"compressor":{"id":"zlib","level":3},"dtype":"<f8","fill_value":null,"filters":[{"elementsize":8,"id":"shuffle"}],"order":"C","shape":[24],"zarr_format":2}'

The chunks of this netCDF variable are twice compressed with the same method (DEFLATE, also called zlib) but different compression levels (first, 8, second, 3). This is allowed by the HDF5 library but not possible for Zarr data: There can be only one compression method and applied only once. This is confirmed in the .zarray metadata where the shuffle and the second DEFLATE (zlib) compression are recorded, but not the zlib level 8. When the chunk buffer reaches the Kerchunk’s Shuffle codec, it is still not decompressed to its original size and, hence, the ValueError you are seeing.
Regenerating that file with one of the NCO tools probably applies only one zlib compression and thus makes those chunks acceptable for the Zarr software. You can check this by running the h5dump command for the same netCDF variable on the ncks-generated file. -> I did check this and confirmed it. The h5dump for the file that came out of ncks showed:
   FILTERS {
      PREPROCESSING SHUFFLE
      COMPRESSION DEFLATE { LEVEL 3 }
   }

Thanks very much to @ajelenak for untangling a really mysterious error!

Can you find the h5obj.compression values given by h5py for one of the affected variables? You could put a breakpoint in SingleHdf5ToZarr around line 200 if you don't want to do it by hand - or just send me one file.

Would you like to hand-edit your kerchunk output for the broken file to add a "filter" to the .zarray and see if it works?

I changed singles[0]["refs"]["ocean_time/.zarray"] from
'{"chunks":[512],"compressor":{"id":"zlib","level":3},"dtype":"<f8","fill_value":null,"filters":[{"elementsize":8,"id":"shuffle"}],"order":"C","shape":[24],"zarr_format":2}'

to add a filter like in your example:
singles[0]["refs"]["ocean_time/.zarray"] = '{"chunks":[512],"compressor":{"id":"zlib","level":3},"dtype":"<f8","fill_value":null,"filters":[{"elemetsize":8,"id":"shuffle"},{"id": "zlib", "level":3}],"order":"C","shape":[24],"zarr_format":2}'

and then my MultiZarr command worked! However, I did get the same error when I tried to open the dataset using the result of that MultiZarr (see below for basic code) so do I need to change it more than one place maybe?
mzz = MultiZarrToZarr(
    singles,
    concat_dims=["ocean_time"],
 )
out = mzz.translate()

import xarray as xr
ds = xr.open_dataset(
    "reference://", engine="zarr",
     backend_kwargs={
          "storage_options": {
              "fo": out,
          },
          "consolidated": False
    }
)

Is this a different approach

Yes, this would be the route to fixing it in kerchunk so that no manual rewriting of the JSON will be necessary in the future.

However, I did get the same error when I tried to open the dataset using the result of that MultiZarr (see below for basic code) so do I need to change it more than one place maybe?

It its the same across all inputs, I would have expected it to work without anything else, but since I've never seen this before, it's always possible there's another spot that needs changing.

h5obj.compression gives gzip for the variable ocean_time in my affected file. Is that the sort of thing you're looking for?

it's always possible there's another spot that needs changing

That have me the idea to use a preprocess function to change this for all variables with ".zarray" in them and that did the trick.

However zarr DOES allow multiple compressors. There is some confusion in the spec with a false distinction between "filters" and "compressors"

I did not know this, very useful! Perhaps the Zarr spec should state that specifying only filters, including compression filters, is enough?
Fixing this in kerchunk will require using more low-level h5py API. That h5py.Dataset.compression property will not do it for cases like this one.

I did not know this, very useful! Perhaps the Zarr spec should state that specifying only filters, including compression filters, is enough?

Perhaps, but I assume that the language around this is more exact in V3, where all of the current focus is.

Fixing this in kerchunk will require using more low-level h5py API.

Is this difficult?

Fixing this in kerchunk will require using more low-level h5py API.

Is this difficult?

I am not the right person to answer this question. 😄
What needs to be done is very similar to this h5py code: https://github.com/h5py/h5py/blob/6b5af4c6495bf865fee5f036122187c21fcb17d4/h5py/_hl/filters.py#L294-L331.

Loop over all declared filters and translate to the numcodecs equivalents.
Recognize which ones are compression filters.
Declare the last compression filter as the compressor in the Zarr array's metadata and put all the other filters in the Zarr array's metadata as filters list.

The above may be simpler for Zarr v3 where, I think, the same approach as in HDF5 was taken that all filters are treated equally but are called "codecs".

Recognize which ones are compression filters.

Is it not the case that all HDF filters are bytes->bytes?

Yes. I was thinking of the need to decide which of the compressors (if more than one) should be declared in the Zarr array metadata as such.

Yes. I was thinking of the need to decide which of the compressors (if more than one) should be declared in the Zarr array metadata as such.

For v2, I believe it's fine to have multiple filters and NO compressor. When decoding, codecs are called last-first. IF there is a compressor, it simply becomes the last filter, and gets called first.


Add Toggle for Elf (Elven Ears)

Add a toggle or otherwise option to display the Mercedes ears.

Created PR https://github.com/crrio/maplestory.design/pull/63 with that feature

Added with 6a21248567e930c2d9c24452e81614ee325a1059


IdentifierStrategyListener throws an exception if the execute scalar result is null which can happen if you have a de-duping index and submit a duplicate record...

It is possible to make code which inserts records into a table idempotent by adding a unique index on the table which ignores duplicates. If you do this, MicroLite throws an exception because it expects the table PK to be returned.


filter clause with where = or = fail or are slow

Queries with filter where 1=1 will fail. Several of the optimizers hit a null pointer exception. I tried tracking these down, but kept hitting ever more complex null pointer issues.
Queries with filter where "colA" = "colA" work correctly, but for expensive queries, these end up taking almost twice as long.
While this is quite a weird edge case, we see it where users are passing a string query between functions. They start the query with where A=A so they can keep appending filters after that without need to know if there was already a filter.

The reason why where colA = colA is slow is because it will be converted to where colA - colA = 0, which involves transform.
One solution is to remove the equality predicate when the LHS and RHS are the same.


Whitespace Normalization by Configuration

Currently all whitespace of the parsed IMSC XML document are retained.  This leads to irritating displays in the input fields for text where text has leading and trailing spaces.

But most of the time leading and trailing whitespaces are not significant and could be trimmed.

Therefore there should be a configuration option to whitespace normalize the incoming XML document. It may be sufficient to trim trailing spaces of the content elements div, p and span.

The current implementation now trims the begin and end of whitespace for the view in the editor. But if the subtitle is not edited the whitespace will be exported unchanged. If the subtitle is edited the whitespace gets also trimmed in the export. This may be difficult for users to understand and may need to be made more consistent.

Another current problem is, that when a <span> element contains essential whitespace (e.g. to separate words), it depends on the editing action if this kept or not. This behavior points to a bug.


Update dependencies

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (693c446) to head (f2e8275).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #780   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           90        90
  Lines         2459      2459
  Branches       163       163
=========================================
  Hits          2459      2459

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Implement Middleware for Either

One usage in context:
let cors = if config.cors_enabled {
    let cors = CorsBuilder::new()
        .allow_origins(AllowedOrigins::Any { allow_null: true })
        .allow_headers(vec![header::CONTENT_TYPE])
        .allow_methods(vec![Method::GET, Method::POST])
        .allow_credentials(false)
        .max_age(Duration::from_secs(ONE_HOUR_IN_SECONDS as u64))
        .prefer_wildcard(true)
        .build();

    Either::A(cors)
} else {
    Either::B(Identity::new())
};

ServiceBuilder::new()
    .resource(Index::new(config.root.clone()))
    .resource(Assets::new(config.root))
    .resource(SandboxFixme)
    .resource(Meta::default())
    .resource(Gist::new(config.gh_token))
    .middleware(cors)
    .run(&addr).unwrap();

Could even provide a helper thing:
let m = middleware::maybe(some_bool, || {
    // create middleware here
})

a helper thing:

I say this, but as far as I can tell you either leak the type:
fn maybe<M>(enabled: bool, f: impl FnOnce() -> M) -> Either<M, Identity> {
    if enabled {
        Either::A(f())
    } else {
        Either::B(Identity::new())
    }
}

Or the trait bounds for impl trait are so complicated that I have not yet figured them out.

I can merge this now but sooner then later, Middleware will be moved into a dedicated crate and at which point you will no longer be able to impl Middleware for either due to coherence.
Either way, this solution also isn’t fully generic to all Middleware (only http) so that means there is still a problem...

Middleware will be moved into a dedicated crate

tower-web requires M: HttpMiddleware; so this tailored for that case. Do you expect this new crate to be beyond tower-web? Will there be a http-middleware specific crate if so?
Would you prefer there to be a custom EitherHttpMiddleware type?

I expect that there will be tower-http for all the http specific things.
I wonder if we could have an IntoEither trait for “joining” the response
type from two either branches. The problem would then be where does the
implementation for http::Response live.
On Tue, Aug 28, 2018 at 8:57 AM Jake Goulding<EMAIL_ADDRESS>wrote:

Middleware will be moved into a dedicated crate
tower-web requires M: HttpMiddleware; so this tailored for that case. Do
you expect this new crate to be beyond tower-web? Will there be a
http-middleware specific crate if so?
Would you prefer there to be a custom EitherHttpMiddleware type?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/carllerche/tower-web/pull/81#issuecomment-416640304,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAYJOYFOLR-SoNYWhn83SHtXtFfAy17ks5uVWhpgaJpZM4WMcEI
.


Add support for HRF convolution

For a variety of reasons, it makes sense to move HRF convolution out of fitlins and into pybids. This will make it much easier to construct "final" design matrices we can just hand off to estimators like nistats. We can achieve this very easily by bundling @bthirion's hemodynamic_models module from nistats in pybids.
@bthirion, any objections? We could also add nistats as a dependency, but I don't think it makes sense to do that at the moment given that this is the only thing we need from nistats, and I'd like to avoid adding sklearn and nilearn as dependencies if we can help it.

pybids doesn't do any data processing; right now it never even reads an image (well, except to determine the number of volumes). I'd like to keep it that way if I can help it... :)

Would it make sense to just make the design matrix bits a small, self-contained package that pybids and nistats could each depend on? It should have an order of magnitude less churn than the rest of nistats, as it's (almost) exactly the same as what was in nipy, before.

I'd rather not: we realize that package maintenance is a burden, and we prefer to have fewer, well maintained packages rather than a bunch of small ones that nobody will take care of.
For this we reason, we'll actually merge Nistats into Nilearn in the next months (Yes, I've been saying that for months, but now we have somebody actively working on it) !