Emitting Type Annotations for global variable

Hello Adrian,
I am trying to emit type annotations for global pointer. This is how my C code looks like -
int CHECK_AV *  shd;

int main() {
  shd = (int*)malloc(10 * sizeof(int));
  return 0;
}

This is how my llvm bitcode looks like -
define dso_local i32 @main() local_unnamed_addr #4 {
entry:
  %call = tail call noalias i8* @malloc(i64 40) #3
  store i8* %call, i8** bitcast (i32** @shd to i8**), align 8, !tbaa !2 // I want to add type annotation here
  ret i32 0
}

tbaa annotation is coming from  this call CGM.DecorateInstructionWithTBAA(Store, TBAAInfo); from void CodeGenFunction::EmitStoreOfScalar. That's why I  have also added  CGM.TADecorate(Store, Ty);  there.
I am printing this comment in CGTypeAnnotation.cpp  -
  std::cout<<"success1\n";
  Inst->setMetadata("tyann", node);

and I am getting success printed out while compilation but tyann is not attached to IR. I am new to clang, so not able to figure out how should I debug.
Amazing thing is if I change my code to
int main() {
  shd = (int*)malloc(10 * sizeof(int));
  shd[1]++;
  return 0;
}

I get my annotations from call from EmitStoreOfScalar
; Function Attrs: norecurse nounwind uwtable
define dso_local i32 @main() local_unnamed_addr #4 {
entry:
  %call = tail call noalias i8* @malloc(i64 40) #3
  store i8* %call, i8** bitcast (i32** @shd to i8**), align 8, !tbaa !2
  %arrayidx = getelementptr inbounds i8, i8* %call, i64 4
  %0 = bitcast i8* %arrayidx to i32*
  %1 = load i32, i32* %0, align 4, !tyann !6
  %inc = add nsw i32 %1, 1
  store i32 %inc, i32* %0, align 4, !tbaa !7, !tyann !6
  ret i32 0
}

If you can help me, it will mean a lot to me.

Interesting! So it sounds like the type qualifier isn’t attached to the type of the Store value at that point in codegen.
My first guess is that this is because the store you’re looking at is managing the pointer to a type-qualified value; not the value itself. The pointer type is not qualified; just the pointed-to type. This is usually a nice distinction to have—for example, in an information flow system, you might have a private pointer to public data.
But if it’s getting in your way, you might consider adding a simple check to detect when the result of a malloc call is later dereferenced to a qualified value with load.

Thank you so much for your response. I am only concerned with type annotated value. It might happen that I pass this pointer to another function and that function reads and writes value pointed by this pointer but in this function, this value won't be annotated. For this reason, I need pointer type to be qualified. How can I do that? Any suggestions?

Well, before diving into that, it might make sense to verify my hypothesis above. For example, take a look at the test input for the included nullability checker:
https://github.com/sampsyo/quala/blob/master/examples/nullness/test/simple.c
It looks like I figured out the right way to annotate the pointer, rather than the int, in a declaration was int * NULLABLE b;, which is the opposite of your example. Maybe the solution is as simple as that reordering?

Program-
int * CHECK_AV shd;

int main() {
  shd = (int*)malloc(10 * sizeof(int));
  return 0;
}

when I compile with -O0, I do get annotation
define dso_local i32 @main() #5 {
entry:
  %retval = alloca i32, align 4
  store i32 0, i32* %retval, align 4
  %call = call noalias i8* @malloc(i64 40) #3
  %0 = bitcast i8* %call to i32*
  store i32* %0, i32** @shd, align 8, !tyann !2
  ret i32 0
}

but when I compile with -O1, I don't
define dso_local i32 @main() local_unnamed_addr #4 {
entry:
  %call = tail call noalias i8* @malloc(i64 40) #3
  store i8* %call, i8** bitcast (i32** @shd to i8**), align 8, !tbaa !6
  ret i32 0
}

How can I make sure, that I do get same annotations in O1 as I did in O0?

Great!
Preserving metadata through optimizations actually seems like a really hard problem. I’d recommend you run your pass that consumes annotations early in the pipeline instead.

It does not look like as no one has done it before because if you see tbaa annotations, they are preserved through optimizations. Anyways, I will try to figure it out. Thank you so much for your time. I truly appreciate it.

Great! Yeah, if you’re feeling ambitious, it does seem possible with a good chunk of hacking. Good luck, and I’d be interested to hear how it goes!


Android Support

I open this issue just to provide some information about android cross compile:
First thing is that, android-ndk do not have backtrace, hence, we need some workaround like chromium.
Second, there is a little difference of the malloc prototype:
SNMALLOC_EXPORT size_t SNMALLOC_NAME_MANGLE(malloc_usable_size)(const void* ptr)
{
    return Alloc::alloc_size(const_cast<void*>(ptr));
}

With the above changes, I successfully cross compiled for android.
Android (5900059 based on r365631c) clang version 9.0.8 (https://android.googlesource.com/toolchain/llvm-project 207d7abc1a2abf3ef8d4301736d6a7ebc224a290) (based on LLVM 9.0.8svn)
Target: aarch64-unknown-linux-android29
Thread model: posix
InstalledDir: /opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin

It is extremely hard to find a test environment to check whether the allocator is really working, but if you want to have a try, the above workarounds can be useful.

I'm going to leave this issue open until we have CI support to test Android support.

Preliminary support provided by #171.

https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/android?view=azure-devops

Is there more we should do here?


Version Tracking on two copies of the same plain-jane Visual Studio 2022 "Hello World" Windows C++ Console App causes MemoryAccessException upon Automatic Version Tracking.

Describe the bug
Version Tracking on two copies of the same plain-jane Visual Studio 2022 "Hello World" Windows C++ Console App causes MemoryAccessException upon Automatic Version Tracking.
To Reproduce
Steps to reproduce the behavior:

Install/start Visual Studio 2022 Community Edition.
Create New Project, Windows C++ Console App, click Next, project name is SimpleConsoleApp, select Location, check "Place solution... in the same directory", click Create.
With boilerplate comments removed, the following default Hello World should remain as follows...

#include <iostream>
int main()
{
    std::cout << "Hello World!\n";
}

With Debug/x64 selected, Build/Build Solution.
Right-click SimpleConsoleApp project, choose Open Folder in File Explorer, navigate to .\SimpleConsoleApp\x64\Debug where the .exe and .pdb are located.
Select both .exe/.pdb, use Copy/Paste to duplicate them (i.e., select both, Ctrl+C, Ctrl+V), then rename the copies to be SimpleConsoleApp2.exe and SimpleConsoleApp2.pdb. All these files should be in the same .\Debug output folder. These represent two binaries which are equal. We will be create a Ghidra Version Tracking session using SimpleConsoleApp.exe and SimpleConsoleApp2.exe.
Start Ghidra.
File/New Project, Non-Shared Project, Project Name is SimpleConsoleAppVtProj, click Finish.
Press I to import, locate and select SimpleConsoleApp.exe, click Select File To Import, click OK on the next Import dialog, click OK on the Import Results Summary dialog.
Repeat step 9 but for SimpleConsoleApp2.exe.
Highlight both SimpleConsoleApp.exe and SimpleConsoleApp2.exe, right-click and choose Open With, Version Tracking, SimpleConsoleApp.exe is the Source Program, SimpleConsoleApp2.exe is the Destination Program, click Next, click Run Precondition Checks, click Next, click Finish...
Do not choose to auto-analyze, instead click the dialog box X or No to skip auto-analysis for both binaries. This is done simply to go incrementally, where it is desired to manually locate/load the .pdb first then auto-analyze, not that this is required but these are the steps taken. It also confirms symbols were loaded properly fwiw.
Ghidra VT will open both binaries in analysis windows... in the SimpleConsoleApp.exe window's menu choose Edit/Symbol Server Config, click +, Directory, specify the path to the SimpleConsoleApp.exe pdb file, click OK, click OK.
File/Load PDB File..., click Load, it should load, click OK to dismiss warnings.
Press A to auto-analyze SimpleConsoleApp.exe.
Repeat steps 14 and 15 for SimpleConsoleApp2.exe.
Choose to Save All for both analysis windows.
Go to Version Tracking Window, File/Automatic Version Tracking, observe the following exception...

ghidra.program.model.mem.MemoryAccessException: Couldn't get all bytes for CodeUnit
java.lang.RuntimeException: ghidra.program.model.mem.MemoryAccessException: Couldn't get all bytes for CodeUnit
	at ghidra.app.plugin.match.MatchData.extractSourceHashes(MatchData.java:83)
	at ghidra.app.plugin.match.MatchData.matchData(MatchData.java:47)
	at ghidra.feature.vt.api.correlator.program.DataMatchProgramCorrelator.doCorrelate(DataMatchProgramCorrelator.java:61)
	at ghidra.feature.vt.api.util.VTAbstractProgramCorrelator.correlate(VTAbstractProgramCorrelator.java:73)
	at ghidra.feature.vt.gui.actions.AutoVersionTrackingTask.correlateAndPossiblyApply(AutoVersionTrackingTask.java:587)
	at ghidra.feature.vt.gui.actions.AutoVersionTrackingTask.doRun(AutoVersionTrackingTask.java:193)
	at ghidra.feature.vt.gui.actions.AutoVersionTrackingTask.run(AutoVersionTrackingTask.java:123)
	at ghidra.util.task.Task.monitoredRun(Task.java:134)
	at ghidra.util.task.TaskRunner.lambda$startTaskThread$0(TaskRunner.java:106)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: ghidra.program.model.mem.MemoryAccessException: Couldn't get all bytes for CodeUnit
	at ghidra.program.database.code.CodeUnitDB.getBytes(CodeUnitDB.java:653)
	at ghidra.app.plugin.match.MatchData.extractSourceHashes(MatchData.java:80)
	... 11 more

---------------------------------------------------
Build Date: 2024-Jan-30 1212 EST
Ghidra Version: 11.0.1
Java Home: <...>
JVM Version: Eclipse Adoptium 17.0.10
OS: Windows 11 10.0 amd64

Expected behavior
Generally, a successful no-exception automatic version tracking completion would be expected regardless of any match status.
Screenshots
N/A, see steps/exception above.
Attachments
N/A, see steps/exception above.
Environment (please complete the following information):

OS: Windows 11 22H2
Java Version: Eclipse Adoptium 17.0.10
Ghidra Version: 11.0.1
Ghidra Origin: official GitHub distro

Additional context
This issue was encountered when trying to create a simple example for yet another Version Tracker bug causing a different exception. I hit the above instead, removed my test code for the other issue, and discovered this issue's exception with a just a simple "hello world" app (which pulls in RTL, C++ STL but relatively simple).

It seems DataMatchProgramCorrelator is not handling uninitialized memory properly as does other parts of Ghidra. The DataMatchProgramCorrelator fails on the same exception that CodeBrowser ignores when a user is scroll across the same memory. DataMatchProgramCorrelator should view two equal uninitialized areas as equal.
Details...

At the exception stack of the original post of this issue (above)...
The initial address during the relevant getBytes call is Address.offset ==<PHONE_NUMBER> which is 0x14001D1E0.
The getBytes request needs a total of 152 bytes.
The MemoryBlockDB instances (as assessed during MemoryMapDB.getBlockDB)...

tmpBlocks	(id=449)
    [0]	MemoryBlockDB  (id=571)
    [1]	MemoryBlockDB  (id=572)
    [2]	MemoryBlockDB  (id=439)
    [3]	MemoryBlockDB  (id=573)
    [4]	MemoryBlockDB  (id=379)
        adapter	MemoryMapDBAdapterV3  (id=441)
        id	4
        invalid	false
        lastSubBlock	FileBytesSubMemoryBlock  (id=386)
        length	512
        mappedBlocks	null
        memMap	MemoryMapDB  (id=372)
        record	DBRecord  (id=583)
        startAddress	GenericAddress  (id=585)
        subBlocks	ArrayList<E>  (id=451)
            [0]	FileBytesSubMemoryBlock  (id=386)	// 32 bytes satisfied here, ok.
    [5]	MemoryBlockDB  (id=389)
        adapter	MemoryMapDBAdapterV3  (id=441)
        id	5
        invalid	false
        lastSubBlock	UninitializedSubMemoryBlock  (id=460)
        length	1856
        mappedBlocks	null
        memMap	MemoryMapDB  (id=372)
        record	DBRecord  (id=593)
        startAddress	GenericAddress  (id=594)
        subBlocks	ArrayList<E>  (id=481)
            [0]	UninitializedSubMemoryBlock  (id=460)   // uninitialized, cannot satisfy the remaining 120 bytes.

The getBytes request satisfies 32 bytes at the MemoryBlockDB of index 4.
It fails trying get the remaining 120 bytes from the UninitializedSubMemoryBlock at index 5.
Following is what CodeBrowser shows at the start address of the failing getBytes request. There are 32 valid bytes at 14001d1e0 followed by a gap and uninitialized memory.

14001d1e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 _EXCEPTION_RECORD
          00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
14001d278 ??
14001d279 ??

As you have mentioned in your follow on, it looks like the Data correlator is the culprit whether or not called from Auto VT. Thanks for tracking down the underlying issue.

Would you mind attaching the two files so we can see exactly what you are working with? It is not easy to get the same exact binary without knowing all of your compile settings.

I prefer to avoid sharing physical binaries publicly online if reasonable. In this case, it's tremendously easy to recreate the binaries using Visual Studio 2022 Community Edition with its default Windows C++ Console App project. Would you be able to do that? (See steps in prior post and let me know.)

Would you mind attaching the two files so we can see exactly what you are working with? It is not easy to get the same exact binary without knowing all of your compile settings.

I prefer to avoid sharing physical binaries publicly online if reasonable. In this case, it's tremendously easy to recreate the binaries using Visual Studio 2022 Community Edition with its default Windows C++ Console App project. Would you be able to do that? (See steps in prior post and let me know.)

No worries. We will do so and hopefully our setup will build the exact same ones or similar enough ones. Thanks.

I dove in a little further after yesterday's update... here are some additional details fwiw...
Summary
An affected PE includes both .data and .bss within the physical PE .data section, where portions of .bss (uninit data) can reside in the initialized portion of the .data section. Portions of .bss beyond the physical .data section size (virtualSize-rawDataSize in details below) is deemed "uninitialized" by Ghidra, yet those blocks may not include all of the logically uninitialized data, a byproduct of physical PE construction and Ghidra's section-to-block processing.
Details...
Upon .exe import, ghidra.app.util.opinion.PeLoader.processMemoryBlocks(...) is called...
Daemon Thread [Task - Import File] (Suspended)
	PeLoader.processMemoryBlocks(PortableExecutable, Program...) line: 673
	PeLoader.load(...) line: 125
	PeLoader(...).doLoad(...) line: 883
	PeLoader(...).loadProgram(...) line: 98
	PeLoader(...).load(...) line: 131
	ImporterUtilities.importSingleFile(...) line: 395
...

The processMemoryBlocks function processes each PE section, using virtualSize and rawDataSize to guide creation of initialized/uninitialized memory blocks.
For example, a .text section might have the following values...
sectionName ... ".text" (id=669)
addr ..........<PHONE_NUMBER> [0x140001000]
virtualSize ... 4608 [0x1200]
rawDataSize ... 4608 [0x1200]

The rawDataSize is deemed initialized data contained within the PE's section itself.
Any amount of virtualSize that is above rawDataSize is deemed uninitialized data.
Therefore, for each PE section where rawDataSize >0 bytes, processMemoryBlocks calls MemoryBlockUtils.createInitializedBlock to create an initialized data memory block.
For the above .text section example, since rawDataSize==virtualSize  (4608==4608), no uninitialized block is created for the .text section.
In the case reproducing the issue, the PE's physical .data section includes both the logical .data and .bss (uninitialized data) sections. I coin "logical" here to emphasize that .bss is a section to the developer concerned with link-time behaviors/outcomes so .bss exists as a section for uninitialized data in that sense, yet such is not reflected physically in the PE causing the issue to arise.
From the same example PE reproducing the issue, here are the values for its .data section...
sectionName ... ".data" (id=696)  // includes .bss
addr ..........<PHONE_NUMBER> [0x140004000]
virtualSize ... 1011 [0x3f3]      // larger than area in PE
rawDataSize ... 512 [0x200]       // area in PE

Note that virtualSize is larger than rawDataSize.
Given the above, processMemoryBlocks calculates 499 bytes for uninitialized block size...
rawDataSize == 512 == initialized block size (includes portions of .bss data).
(virtualSize-rawDataSize) == Ghidra uninitialized block size.
1011 - 512 == 499 bytes for Ghidra uninitialized block size.

For the above .data section example, processMemoryBlocks calls MemoryBlockUtils.createInitializedBlock to create a 512 byte initialized data segment (which technically includes a portion of uninitialized data), and it follows up with a call to MemoryBlockUtils.createUninitializedBlock to create a 499 byte uninitialized block.
Perhaps a helpful way to view all this... there is no penalty beyond impact to the binary size by actually initializing uninitialized data with zeros. Therefore, there is no penalty (beyond binary size) for the linker to include a portion of .bss in the initialized section, which is what happens in the example.
So the .data (initialized data) section can include a portion of uninitialized data which is, in fact, initialized to zero. The implications to Ghidra's current section/block processing are that uninitialized data items may span what Ghidra views as both initialized and uninitialized blocks.
Given the above example, the following is derived from Ghidra's disassembly. Notice .data begins at 0x140004000 and 0x200 bytes are included in the PE, yet within that 0x200 is a portion of .bss (uninit data)...
                //
                // .data
                // ram:140004000-ram:1400041ff
                //
                .data
                normal_data1
140004000  01                   uchar  01h
                normal_data2
140004001  02                   uchar  02h
                normal_data3
140004002  03                   uchar  03h
140004003  00                   ??     00h
...
14000410f  00                   ??     00h
                        .bss
                        my_bss
140004110  00  00  00  00  00  00  00  00  00      MYSTRUCT
           00  00  00  00  00  00  00  00  00
           00  00  00  00  00  00  00  00  00...
140004110  00   uchar     '\0'  field1
140004111  00   uchar     '\0'  field2
140004112  00   uchar     '\0'  field3
140004113  00   uchar     '\0'  field4
140004114  00   ??        00h   field4_0x4
140004115  00   ??        00h   field5_0x5
140004116  00   ??        00h   field6_0x6
...
1400041fd  00   ??        00h   field237_0xed
1400041fe  00   ??        00h   field238_0xee
1400041ff  00   ??        00h   field239_0xef
140004200       ??        ??    field240_0xf0
140004201       ??        ??    field241_0xf1
140004202       ??        ??    field242_0xf2
...
1400042f8       ??        ??    field488_0x1
1400042f9       ??        ??    field489_0x1
1400042fa       ??        ??    field490_0x1
1400042fb       ??        ??    field491_0x1
1400042fc       ??        ??    field492_0x1
1400042fd       ??        ??    field493_0x1
1400042fe       ??        ??    field494_0x1
1400042ff       ??        ??    field495_0x1
140004300       uint64    ??    array
140004308       ??        ??    field497_0x1
140004309       ??        ??    field498_0x1
14000430a       ??        ??    field499_0x1
...
140004375       ??        ??    field606_0x2
140004376       ??        ??    field607_0x2
140004377       ??        ??    field608_0x2

Relating to the details of my previous post, during Automatic Version Tracking, the VT correlation ends up calling into ghidra.program.database.mem.MemoryMapDB.getBytes(...) ...
Daemon Thread [Task - Auto Version Tracking Command] (Suspended)
    MemoryMapDB.getBytes(Address, byte[], int, int, boolean) line: 1439
    MemoryMapDB.getBytes(Address, byte[], boolean) line: 1432
    DataDB(CodeUnitDB).getBytes() line: 652
    MatchData.extractSourceHashes(Program, AddressSetView, ...) line: 80
    MatchData.matchData(Program, AddressSetView, Program, AddressSetView, ...) line: 47
    DataMatchProgramCorrelator.doCorrelate(VTMatchSet, ...) line: 61
    ...

The getBytes function sees the initialized block, is able to get the bytes, but hits the uninitialized block on the second iteration to complete the getBytes request.
For the heck of it, I tried out the following quick test which allows auto VT to complete by allowing the caller of getBytes (the correlator in this case) to request that uninitialized blocks be treated as zero bytes...
public int getBytes(Address addr, byte[] dest, int dIndex, int size, boolean zeroBytesForUninitialized)
        throws MemoryAccessException {
    ...
    int numRead = 0;
    long lastRead = 0;
    while (numRead < size) {
        try {
            addr = addr.addNoWrap(lastRead);
            MemoryBlock block = getBlock(addr);
            ...
            int destOffset = numRead + dIndex;
            int numToRead = size - numRead;
            if (block.isInitialized() || block.isMapped()) {
                lastRead = block.getBytes(addr, dest, destOffset, numToRead);
            }
            else if (!block.isInitialized() && zeroBytesForUninitialized) {
                Arrays.fill(dest, destOffset, destOffset + numToRead, (byte)0);
                lastRead = numToRead;
            }
            else {
                break;
            }
            numRead += lastRead;
        }
        ...
    }
    ...
    return numRead;
}

One downside to assuming zero bytes for uninitialized data would be a case where a previously uninitialized area is later modified to be initialized to zeros which would then hide the change itself from the correlator making the "zero-byte" assumption. It seems the best for correlation to consider uninit data to be only equal to contextually matching uninit data to mitigate a correlator from missing changes relevant to what it should detect (i.e., a prev unint area changed to an init area of same size should be a detected change, it seems).

Would you mind attaching the two files so we can see exactly what you are working with? It is not easy to get the same exact binary without knowing all of your compile settings.

I prefer to avoid sharing physical binaries publicly online if reasonable. In this case, it's tremendously easy to recreate the binaries using Visual Studio 2022 Community Edition with its default Windows C++ Console App project. Would you be able to do that? (See steps in prior post and let me know.)

No worries. We will do so and hopefully our setup will build the exact same ones or similar enough ones. Thanks.

With the current build tools, Visual Studio 2022 Community Edition, it is extraordinarily easy to reproduce this issue.
I would be surprised if you found a way to build a simple .exe using typical steps that would avoid the issue. I say this because of what's mentioned above... so long as you create an .exe that has a .data section whose binary data size is smaller than its virtual size, you should hit this issue. Generally, I see .data sections with 0x200 bytes in the PE and a virtual size larger than that 0x200 bytes... this is what causes the init/uninit blocks to be created, if that makes sense.
...but definitely let me know if anything is not easy to reproduce.

If you want something more minimal, the following may be helpful/interesting...
You can use a .asm and Visual Studio 2022's ml64.exe to build a minimal .exe that may help remove noise and see a simpler layout. The example PE sections of my previous post were from a PE created with the following steps.
Steps:

Install/start Visual Studio 2022 Community Edition on Windows 11. (Windows 10 is likely fine too.)
Press Start Menu ("Super Key" in Linux lingo), type "2022" to show Visual Studio 2022 command line shortcuts, click/select "x64 Native Tools Command Prompt for VS 2022" to open an x64 tools prompt (aka x64 native tools terminal window).
Optionally, within the new terminal window, type "powershell" and press ENTER if you prefer PowerShell.
Create a new .asm source text file, Simple.asm, and copy/paste the following source code into that file...

; Build: ml64 /Fl /Zi .\Simple.asm /link /subsystem:console /entry:main /DEBUG

CONST SEGMENT PARA 'const'
sz_howdy_world db "Howdy there, world!",0
CONST ENDS

MYSTRUCT STRUCT 16
    field1 db ?
    field2 db ?
    field3 db ?
    field4 db 490 DUP(?)
    array dq 15 DUP(?)
MYSTRUCT ENDS

.DATA
normal_data1 db 1
normal_data2 db 2
normal_data3 db 3

.DATA?
my_bss MYSTRUCT <>

.CODE
main PROC
    sub rsp,28h                                     ; Allocate shadow space
    mov rax,offset normal_data1
    mov rax,offset normal_data2
    mov rax,offset normal_data3
    mov rax,offset my_bss
    mov BYTE PTR [rax],0aah
    add rsp,28h  ; Deallocate shadow space
    xor eax,eax                                     ; exit code is 0 success.
    ret
main ENDP
END

Run ml64 /Fl /Zi .\Simple.asm /link /subsystem:console /entry:main /DEBUG to assemble/link to PE .exe. In the same directory, you will now see Simple.exe and you may want to gander at Simple.lst, the assembly listing file.

The remainder of the steps are essentially the same as those in my original post...

Copy Simple.exe to Simple2.exe or some other name to create to versions. Differences are not important, they can match 100% and the issue is still observable. Alternatively, copy Simple.asm to Simple2.asm and build that to create the other binary. Goal is obviously to have two distinct physical binaries.
Start Ghidra.
File/New Project, Non-Shared Project, Project Name is SomeVtProj, click Finish.
Press I to import, locate and select Simple.exe, click Select File To Import, click OK on the next Import dialog, click OK on the Import Results Summary dialog.
Repeat step 9 but for Simple2.exe
Open Simple.exe in the CodeBrowser, from the CodeBrowser's menu choose Edit/Symbol Server Config, click +, Directory, specify the path to the Simple.exe pdb file, click OK, click OK. (It is assumed Simple.pdb and Simple2.pdb are in the same directory.)
From the same CodeBrowser menu, use File/Load PDB File..., click Load, it should load Simple.pdb, click OK to dismiss warnings.
Close that CodeBrowser.
At this point, you can open each of Simple.exe and Simple2.exe if you wish to perform analysis ahead of time instead of when Version Tracking opens the CodeBrowser windows. Either way, ensure analysis is complete prior to actually starting Automatic Version Tracking.
Highlight both Simple.exe and Simple2.exe, right-click and choose Open With, Version Tracking, Simple.exe is the Source Program, Simple2.exe is the Destination Program, click Next, click Run Precondition Checks, click Next, click Finish...
Choose to perform any auto-analysis if it was not already done. Do this for both Simple.exe and Simple2.exe.
Go to Version Tracking Window, File/Automatic Version Tracking, observe the following exception...
ghidra.program.model.mem.MemoryAccessException: Couldn't get all bytes for CodeUnit. (See original post for full stack.)

If interested, from the same Visual Studio x64 native tools terminal window, you can use dumpbin to see the section details.
For example, the following shows the .text section as having a raw size larger than its virtual size, whereas the physical .data section (inclusive of .bss) has a raw size of only 0x200 bytes while its actual (virtual) size is 0x3F3 bytes. The uninitialized my_bss struct of the above .asm file spans from the PE's 0x200 initialized area to what will eventually become (upon loading) the uninitialized area of the overall physical .data section...
dumpbin /HEADERS Simple.exe
...
SECTION HEADER #1
   .text name
    1057 virtual size
    1000 virtual address<PHONE_NUMBER>001000 to<PHONE_NUMBER>002056)
    1200 size of raw data
     400 file pointer to raw data (00000400 to 000015FF)
       0 file pointer to relocation table
       0 file pointer to line numbers
       0 number of relocations
       0 number of line numbers
60000020 flags
         Code
         Execute Read
...
SECTION HEADER #3
   .data name
     3F3 virtual size
    4000 virtual address<PHONE_NUMBER>004000 to<PHONE_NUMBER>0043F2)
     200 size of raw data
    1A00 file pointer to raw data (00001A00 to 00001BFF)
       0 file pointer to relocation table
       0 file pointer to line numbers
       0 number of relocations
       0 number of line numbers
C0000040 flags
         Initialized Data
         Read Write
...

Okay, likely not needed, but if you are interested, here is a very simple minimal example that will pass or fail Automatic Version Tracking (AVT) depending on whether you build it with /DFAILING_CASE or not. The difference is 1 byte of initialized data that, if present, causes the first .bss variable to start at the beginning of the .data section beyond the raw data included within the PE.
; Build for success case:
;   ml64.exe /Zi .\SimplePassFail.asm /link /subsystem:console /entry:main /DEBUG
; Build for failing case:
;   ml64.exe /DFAILING_CASE /Zi .\SimplePassFail.asm /link /subsystem:console /entry:main /DEBUG

IFDEF FAILING_CASE
NUM_FILL_BYTES EQU 200
ELSE
NUM_FILL_BYTES EQU 201
ENDIF

MYSTRUCT STRUCT 16
    field1 db ?
    field2 db ?
    field3 db ?
    field4 db 490 DUP(?)
    array dq 15 DUP(?)
MYSTRUCT ENDS

.DATA
fill_var db NUM_FILL_BYTES DUP(0)

.DATA?
my_bss MYSTRUCT <>

.CODE
main PROC
    sub rsp,28h                                     ; Allocate shadow space
    mov rax,offset my_bss
    mov BYTE PTR [rax],0aah
    mov rax,offset fill_var
    mov BYTE PTR [rax],0bbh
    add rsp,28h  ; Deallocate shadow space
    xor eax,eax                                     ; exit code is 0 success.
    ret
main ENDP
END

Thank you for the detailed instructions...I was able to follow your original post and reproduce the issue with VS2022.  All of our loaders create uninitialized blocks for memory that is not backed by the original file.  This is done for reasons of efficiency....some of them can be quite large.  This is not how real memory works of course...there is no concept of "uninitialized" there.  But, we also don't want to precisely model all of memory...or we'd need a lot of memory.  So, as you've noticed, all of our code that works with memory needs to account for both initialized and uninitialized blocks.

Thank you for the detailed instructions...I was able to follow your original post and reproduce the issue with VS2022. All of our loaders create uninitialized blocks for memory that is not backed by the original file. This is done for reasons of efficiency....some of them can be quite large. This is not how real memory works of course...there is no concept of "uninitialized" there. But, we also don't want to precisely model all of memory...or we'd need a lot of memory. So, as you've noticed, all of our code that works with memory needs to account for both initialized and uninitialized blocks.

Thank you @ryanmkurtz (and @ghidra007). What you're saying makes great sense.
If it helps temper prioritization, this issue is not pressing for myself personally. I was investigating what Ghidra had available in this area but am not blocked by this issue, and I do have the aforementioned workaround as needed.
An aside, not shirking here... I'd have pushed out a first Ghidra PR had path been clear but there seemed many potential avenues each affecting internal areas in nuanced ways... seemed best to report, bounce off you.


raise CopierAnswersInterrupt on partial answers

Use case: I'm using the copier api and would like to capture partially completed answers. When completing long questionnaires it's sometimes :sadpanda: to ctrl-c and repeat your work. I plan on:

catching this exception
writing the e.answers.user out to a file.
on subsequent runs, checking for it's existence and populating run_copy(defaults=partial_answers).

I suspect the above procedure could likely be generalized as a first class feature in the project e.g. copier could handle writing a ~/.copier_answers_partial.json  or ~/.coper_{{template_name}}_answers_partial.json, or even write out to SubProject.last_answers.

Nice. But I wonder: if you type a wrong answer, and want to start again to change it, then you can't? because the wrong answer is reused? How will you deal with this?

Almost! See #477. The idea is that you override the template-provided defaults with the dict of previous answers.
Additionally, if you made it partway through and had a few answers to change you can edit the file (see #477 for a snippet that shows the CopierAnswersInterrupt + default_overrides working side by side.

you have the option of editing the answer - it's not skipped.

Ah, of course! OK, that's great then :slightly_smiling_face:

BTW if upstream ever implements https://github.com/tmbo/questionary/issues/81, the problem you're trying to avoid wouldn't even exist. Have you considered it?

Codecov Report

Merging #476 (352d199) into master (9f05fdf) will decrease coverage by 2.42%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master     #476      +/-   ##
==========================================
- Coverage   96.17%   93.74%   -2.43%
==========================================
  Files          39       39
  Lines        2535     2543       +8
==========================================
- Hits         2438     2384      -54
- Misses         97      159      +62

Flag
Coverage Δ

unittests
93.74% <50.00%> (-2.43%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

copier/main.py
74.84% <50.00%> (-19.61%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f05fdf...352d199. Read the comment docs.

BTW if upstream ever implements tmbo/questionary#81, the problem you're trying to avoid wouldn't even exist. Have you considered it?

WRT to this - wasn't aware of this issue, but it doesn't appear to have lots of upstream traction. My concern for this project would be potentially ending up with a redundant way of doing things vis-a-vis questionary.
But in practice, this MR is decoupled from any sort of "resume" functionality in questionary. We're - just tacking on some context to the KeyboardInterrupt. I think the salient thing is maintaining as much compatibility as possible with folks that may already be handling plain-old KeyboardInterrupt so that CopierAnswersInterrupt has a similar shape.
I still imagine regardless of what questionary does, it's not going to suppress the KeyboardInterrupt - it's consumers want a chance to handle that signal as well.


feat: add flags for default execution version

Another go at https://github.com/camunda/camunda-modeler/pull/3528 with added tests and implementations for all editors.
Closes https://github.com/camunda/camunda-modeler/issues/3515 / https://github.com/camunda/camunda-modeler/issues/3696
Ping @nikku FYI
I tried to add the original remarks of @barmac and additionally added configuration for both, cloud and platform versions.
Please let me know what you think.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Kraemer, Daniel seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

@barmac Good point - right now an error is thrown but some fallback logic would be better.
To be fair I must admit that JavaScript is not my natural habitat and a solution would likely not be up to your standards.
So feel free to pick it up from here, happy to learn something new 😉

I pushed changes which implement https://github.com/camunda/camunda-modeler/issues/3515
Regarding default execution platform version if not present on the diagram, I will look into this on Monday.
The error is now swallowed. Probably we could log it easily to developer tools, but having access to application log is a new story.

I am closing this PR in favor of https://github.com/camunda/camunda-modeler/pull/3720. Let me know if this is enough for your use case and feel free to contribute to the discussion on https://github.com/camunda/camunda-modeler/pull/3697#issuecomment-1628560528

@barmac

I gave it a thought in the morning and I am not sure whether we really want to make app behave as if the execution platform version was set to a specific value. That is against the current approach which reflects what is set in the XML. If we implement the requested feature, users won't have incentive to set the platform version on preexisting diagrams.

For us it has shown that users often do not even notice that they have not set the platform version yet - this is why I intended the change in the first place.
The idea would not be to change the current default behavior, of course. But that does not rule out the possibility to make it configurable via flags, does it? I still would like to see this feature added to at least give the chance to have a pre-selection since it still remains the users active choice to have this set or not.
I'm also thinking one should not "override" what ever is set on an existing the diagram, however what if nothing is set on the diagram - at least for me this is still a use case.
So by the end of the day it should be Version set in Diagram > Version set via Flags > Unset.
To explain further on this:
We bundle the modeler together with some (custom) plugins / linter rules and some pre-defined flags alongside some BPMN templates.
These templates currently do not have a "hard-coded" engine version. We could obviously change that, but since we want to use the same templates (=files) for multiple versions of these bundles it is not really desirable either.
These bundles are then given to users that work with the modeler - and especially the templates - to create processes that run against a specific version of the platform.
If we could pre-define this platform version is the bundle right away, there would be no need to adapt the templates for every delivery as well as giving the user the right settings (=flags) for the environment the bundle was intended for.

There is however no obstacle to configure the default version for new diagrams (#2491). This is the only feature I'd merge right now, and leave the other for further discussion.

That's great, thank you already.

As a side note, maybe the execution platform version should not be bound to a diagram but rather to a specific Desktop Modeler instance (or project). As a user, I'd expect linting to follow my cluster's configuration across all of the diagrams. Still, this a change in the app which should be discussed before we start implementing.
We need to make sure what user's problem we want to solve, and tailor the solution to exactly that problem.

I agree 100% - IMHO - the diagrams should not care for the engine version. From a technical perspective they need to care for the modeler version in order to be "compatible" I guess. But making the execution platform bound to the modeler would be what I'm looking for.


Ambient lighting with (optional) SSAO

A full-screen non-directional light that populates the diffuse map with an initial value. Optionally uses a pre-populated SSAO map.
This will require implementing a renderer to produce SSAO maps from G-Buffers.

It turns out that, in practice, applying SSAO to the ambient term is simply not good enough: The diffuse term that follows inevitably obliterates the effect entirely. Instead, the diffuse term should be affected by SSAO, but only very slightly.


Add files via upload

specification has StudyStructure/WorkFlowRef  0-unbounded, but schema has 0-1
specification has StudySummary/StudyParameter 0-unbounded, but schema has 1-unbounded (schema is correct)
specification has StudyParameter/ParameterValue 0-unbounded, but schema has 1-unbounded (schema is correct)
Protocol is not referencing StudyTiming in the schema and also the specification
In schema, StudyTiming is missing DurationTimingConstraint
Specification has StudyInterventions/StudyIntervention 1-unbounded, but schema has 0-unbounded (schema is incorrect)
Specification has StudyObjectives/StudyObjective 0-unbounded, but schema has 1-unbounded (schema is correct)
Specification has StudyEndPoints/StudyEndPoint 0-unbounded, but schema has 1-unbounded (schema is correct)
Specification has StudyEstimands/StudyEstimand 0-unbounded, but schema has 1-unbounded (schema is correct)
Please review cardinalities between children of WorkFlowDef between spec and schema


Add epochs to array broadcasts

Implements in-order delivery for array broadcasts, fixes #3605.
@stwhite91 It seems like AMPI violates the contract to disable anytime migration, namely in the AMPI migration test, migrations were happening concurrently with an array broadcast (which I think was backing an MPI_Scatter), so I enabled anytime migration in AMPI in the first patch of this PR, which fixed the failure. Is there some way AMPI can guarantee when migrations/collectives can happen so that we don't have to use anytime migration for it?
@ZwFink One of the delivery pathways for this is specialized for Charm4py; I think my changes are correct, but it'd be good to get some eyes that are experienced with Charm4py on this.

@stwhite91 It seems like AMPI violates the contract to disable anytime migration, namely in the AMPI migration test, migrations were happening concurrently with an array broadcast (which I think was backing an MPI_Scatter), so I enabled anytime migration in AMPI in the first patch of this PR, which fixed the failure. Is there some way AMPI can guarantee when migrations/collectives can happen so that we don't have to use anytime migration for it?

Yeah, we support AMPI_Migrate_to_pe() but it's not really documented well and isn't used by anyone that I know of. The issue with that test is that we are calling AMPI_Migrate_to_pe on one rank, and then calling MPI_Barrier() for all ranks. MPI_Barrier is currently implemented as a chare array no-op reduction to a broadcast. I'm not sure if we should get rid of support for AMPI_Migrate_to_pe, restrict its usage somehow, or enable anytime migration in AMPI.

@stwhite91 It seems like AMPI violates the contract to disable anytime migration, namely in the AMPI migration test, migrations were happening concurrently with an array broadcast (which I think was backing an MPI_Scatter), so I enabled anytime migration in AMPI in the first patch of this PR, which fixed the failure. Is there some way AMPI can guarantee when migrations/collectives can happen so that we don't have to use anytime migration for it?

Yeah, we support AMPI_Migrate_to_pe() but it's not really documented well and isn't used by anyone that I know of. The issue with that test is that we are calling AMPI_Migrate_to_pe on one rank, and then calling MPI_Barrier() for all ranks. MPI_Barrier is currently implemented as a chare array no-op reduction to a broadcast. I'm not sure if we should get rid of support for AMPI_Migrate_to_pe, restrict its usage somehow, or enable anytime migration in AMPI.

Ah, I see. I think the issue's actually the MPI_Barrier that precedes the AMPI_Migrate_to_pe, delivery of the underlying post-barrier broadcast is racing with the migration.
Are there any benefits for disabling anytime migration other than not having to buffer broadcasts? As far as I know, that's the only optimization that it enables right now (it can't do location optimizations because objects can still migrate via AtSync).

Are there any benefits for disabling anytime migration other than not having to buffer broadcasts? As far as I know, that's the only optimization that it enables right now (it can't do location optimizations because objects can still migrate via AtSync).

That is it, but our current spring cleaning strategy is too simple in that it takes I think 1 minute between calls using a periodic CcdCallback, and if you run the OSU MPI Bcast benchmark on AMPI, the buffered messages can build up quickly and run out of memory in a matter of seconds. We've talked in the past about changing spring cleaning to be aware of the number of messages, the memory consumed by the messages, or decreasing the periodic time limit.

Are there any benefits for disabling anytime migration other than not having to buffer broadcasts? As far as I know, that's the only optimization that it enables right now (it can't do location optimizations because objects can still migrate via AtSync).

That is it, but our current spring cleaning strategy is too simple in that it takes I think 1 minute between calls using a periodic CcdCallback, and if you run the OSU MPI Bcast benchmark on AMPI, the buffered messages can build up quickly and run out of memory in a matter of seconds. We've talked in the past about changing spring cleaning to be aware of the number of messages, the memory consumed by the messages, or decreasing the periodic time limit.

Yeah, it's a bit crude right now and it can definitely get overwhelmed in cases like the ones you're talking about. Maybe the solution for now is to explicitly enable anytime migration only when a user wants to use AMPI_Migrate_to_pe?

Yeah, it's a bit crude right now and it can definitely get overwhelmed in cases like the ones you're talking about. Maybe the solution for now is to explicitly enable anytime migration only when a user wants to use AMPI_Migrate_to_pe?

That sounds good, but wouldn't it be too late by the time AMPI_Migrate_to_pe is called to start buffering bcasts? Or do you mean adding some way for the user to indicate beforehand that they will be using AMPI_Migrate_to_pe?

Yeah, I mean beforehand, at startup or similar. The stablelocations flag gets set when the array is created and there's no interface to change it dynamically as far as I can tell, so I think it'd need to be at the beginning. Besides, the broadcast framework currently doesn't support changing the value of this flag during the execution, so it would likely lead to erroneous deliveries unless it were globally synchronized.

I force pushed a new version that drops the AMPI anytime migration part of the PR now that that's been fixed in #3623. The current PR should be merged via rebase.


Features disappear at zoom > 14 with Leaflet frontend

I am still having an issue described and supposedly solved a couple of years ago: #28. My code:
<script<EMAIL_ADDRESS><script>
    const map = L.map('map')
    let paint_rules = [
              {
                dataLayer:"layername",
                symbolizer:new protomaps.LineSymbolizer({color:"red"})
              }
    ];
    var layer = protomaps.leafletLayer({url:'myfile.pmtiles', paint_rules:paint_rules, maxDataZoom:22})
    layer.addTo(map)
</script>

Behavior: the max zoom is locked at 14. That is, the features added via protomaps (from PMTiles) disappear at zoom levels greater than 14 regardless of me trying to set maxZoom:22 or maxDataZoom:22 (or whatever other value greater than 14).

What is the maxzoom of your myfile.pmtiles? The maxDataZoom should match that, which is likely <= 15, and maxZoom is the maximum display level, for which 22 makes sense.

Ah, I see. When I set both maxDataZoom=14 (which is my data zoom level) and maxZoom=22, all works as intended. Thanks a lot.

Hey guys - in case anyone else gets here, this turns out to be pretty sneaky!
maxDataZoom MUST equal the maxZoom reported in the pmtiles header. You can't just put any ol' number in there, otherwise it will just stop at 14. I spent way too many hours on that.


Initial Angular template

This PR adds resources used throughout the project (bootstrap, jQuery, AngularJS) , establishes a directory structure, and includes a base Angular template that can be revised or expanded upon.

Have you put any thought into using a dependency management tool for bootstrap and angular? Like Bower or NPM? This would simplify getting these kinds of assets and would allow for version management across repos.

@mkinsey You can install the Gulp and Bower programs and  with:
[sudo] npm install -g gulp bower
npm install && bower install

Once dependencies are installed you can launch a test webserver which hosts an example of the angular application.
glup

I used the following resources regarding modularizing angular applications:

http://henriquat.re/modularizing-angularjs/modularizing-angular-applications/modularizing-angular-applications.html
http://briantford.com/blog/angular-bower

@orbitable/frontend Let me know if you have trouble getting started etc.

:+1:


Tools menu options should explicitly state debug/release build

Deploy and Undeploy Tools menu options do not specify whether they affect debug or release build, so one might assume that it follows the current build configuration.  Instead, the menu options always use the debug build.  The menu options should explicitly state debug or release.
This work item was migrated from CodePlex
CodePlex work item ID: '3224'
Assigned to: 'tfabraham'
Vote count: '0'

[UnknownUser@10/9/2008]
Resolved with changeset 16223.

[UnknownUser@2/21/2013]

[UnknownUser@5/16/2013]


Fix installation, and some other bugs

Hi,
This is a cool package! I had trouble getting it installed so I changed some things to make it easier for others.
I created a proper setup.py so that it can be easily installed via pip. For now it can be installed using pip install -e ., but you'll need to modify some things to get it working from any arbitrary dir if you want to have it on pypi.
About the installation:

scikit-image>0.15 do not have a min_size kwarg for skimage.morphology.remove_small_holes() which you use in utils.normalized.clean_image(). So the setup.py is set to work with 0.15.x.
Added __init__.py to model_layers and utils so that they're proper sub-packages that can be identified through setuptools.find_packages()
MANIFEST.in so that network model files etc. are included.

scikit-image 0.15 has the min_size kwarg:
https://scikit-image.org/docs/0.15.x/api/skimage.morphology.html#skimage.morphology.remove_small_holes
it was removed in v0.16:
https://scikit-image.org/docs/0.16.x/api/skimage.morphology.html?highlight=morphology remove_small_holes#remove-small-holes
Bug fixes:

Added usingCL attribute to NuSet class in NuSeT.py. Set it to False.
Set a bunch of variables to have default values of 0 in train_gui.py (line 189-196), otherwise line 272 will raise an exception. Not sure if this is the right way to fix this.

Just to add onto this, there's a bug with tensorflow v1 w/regards to cuDNN and newer RTX cards. The following exception is raised when training or predicting:
2020-08-18 21:49:40.924336: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-08-18 21:49:40.949069: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
  0%|                                                                                | 0/3 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/share/python-virtual-environments/nuset/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/share/python-virtual-environments/nuset/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/share/python-virtual-environments/nuset/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node model_U-Net/conv1-1/Conv2D}}]]
         [[ToFloat/_105]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node model_U-Net/conv1-1/Conv2D}}]]
0 successful operations.
0 derived errors ignored.

This seems to be a well known issue that can be fixed by just setting an environment variable: export TF_FORCE_GPU_ALLOW_GROWTH=true
https://github.com/tensorflow/tensorflow/issues/24496


[Docs] jsx-no-useless-fragment: add more examples of correct code

Hey @ljharb ! 👋
Just a quick PR to:

Add some simpler examples of how to resolve the react/jsx-no-useless-fragment problems when they appear (there are not as many simple examples of just removing the fragment when there's a single child)
Add an example with an embedded expression and ternary

Thanks for the review and merge!


Validate React transformer options earlier

#3537 and #3536 added validation for React JSX transformer options. #3551 fixed a small bug with it.
We should validate options earlier while we still have a mutable copy of ReactOptions, and alter them if invalid. This would allow removing the workaround used in #3551 of storing react_importer again.

Implemented in #3577 and #3578.


Intellisense propose last snippet used instead of first item in list of suggestions

VSCode Version: 1.21.0
OS Version: Windows 10
Steps to Reproduce:

Create two user snippets: A and B
Trigger Intellisense. You are now proposed the list: A, B with A selected as it is ordered that way
Use the snippet B
Trigger Intellisense again. You are proposed with the same list in the same order, but with B as it was the last one you used.

Does this issue occur when all extensions are disabled?: Yes

The problem here is that Intellisense now pre-select the last snippet you have used inside of the list of suggestions. Usually, you return the best suggestions first and you would expect the first item of the list of suggestions to be pre-selected, so that you only press enter and keep on going. This was the previous behavior (version 1.19).
As of now, the ordering that you defined in your extensions is still here, but it is not user friendly anymore since you have now to navigate through the list in order to get the first suggestions. This is kind of breaking the Intellisense experience for users using Snippets when coding.
Thanks,
Quentin

It remember the last selected suggestion, use "editor.suggestSelection": "first", to always select the first suggestion.


Failed to resume download

In my app normally starting and pausing and resuming download works well
but if I repeat  pausing and resuming for several times it fails on resume with unknown error
here is the log info:
D/Main: Resumed download Download(id=2, namespace='Main', url='http://tct-cdn.cdn.p30download.com/p30dl-mobile/Elite.Killer.SWAT.v1.3.3_p30download.com.apk?1520424991', file='/storage/emulated/0/Download/m2m.apk', group=0, priority=NORMAL, headers={}, downloaded=12509168, total=21600554, status=FAILED, error=UNKNOWN, networkType=ALL, created=24982482383666, tag=null)
I appreciate this library but because of this problem i will be forced to think about an alternative.

Ok i will change the link to see if it happens with new link or no
but what can i do to handle this kind of unknown errors? Is it correct to remove download and restart it again or it can be some sort of temporary problems like timeout or something like that and retry will solve it?

@AminDeveloper It defiantly does seem like a timeout issue. What you can do is call fetch.retry(id) for the download to retry at the current position. Keep a counter of the retry attempts. If after lets say 2 attempts then you can stop retrying. I will defiantly look into this a bit further and verify that the library is reporting timeouts correctly.

@AminDeveloper RC11 is now out. Can you test and let me know if you are still getting the UNKNOWN error for the download link?


Bitwise operations

This is a work-in-progress implementation of this CIP, which ultimately aims to solve #4252. I have left the pre-submit checklist in place (below) to be filled in as this gets finished.
Edit: This is now complete. @michaelpj - requesting a review.

I'm getting line-length related blowups here. I used fix-stylish-haskell, but this doesn't seem to address line length issues. In fact, it seems to completely ignore them in some cases, such as imports.
@michaelpj: Am I missing something here? I've literally has fix-stylish-haskell undo some of my line-length-related fixes in the past, and this never used to fail CI before.

I'm also getting utterly bizarre linker errors on MinGW:
/build/ghc124461_0/ghc_1.s: Assembler messages:

/build/ghc124461_0/ghc_1.s:42:0: error:
     Error: CFI instruction used without previous .cfi_startproc
   |
42 |         .cfi_def_cfa rsp, 0
   | ^

/build/ghc124461_0/ghc_1.s:43:0: error:
     Error: CFI instruction used without previous .cfi_startproc
   |
43 |         .cfi_offset rbx, 8192
   | ^

/build/ghc124461_0/ghc_1.s:44:0: error:
     Error: CFI instruction used without previous .cfi_startproc
   |
44 |         .cfi_offset rbp, 8200
   | ^

/build/ghc124461_0/ghc_1.s:45:0: error:
     Error: CFI instruction used without previous .cfi_startproc
   |
45 |         .cfi_offset r12, 8208
   | ^

/build/ghc124461_0/ghc_1.s:46:0: error:
     Error: CFI instruction used without previous .cfi_startproc
   |
46 |         .cfi_offset r13, 8216
   | ^

/build/ghc124461_0/ghc_1.s:47:0: error:
     Error: CFI instruction used without previous .cfi_startproc
   |
47 |         .cfi_offset r14, 8224
   | ^

/build/ghc124461_0/ghc_1.s:48:0: error:
     Error: CFI instruction used without previous .cfi_startproc
   |
48 |         .cfi_offset r15, 8232
   | ^

/build/ghc124461_0/ghc_1.s:49:0: error:
     Error: CFI instruction used without previous .cfi_startproc
   |
49 |         .cfi_offset rip, 8336
   | ^

/build/ghc124461_0/ghc_1.s:50:0: error:
     Error: CFI instruction used without previous .cfi_startproc
   |
50 |         .cfi_escape 0x16, 0x07, 0x04, 0x77, 152, 65
   | ^
`x86_64-w64-mingw32-cc' failed in phase `Assembler'. (Exit code: 1)
make[1]: *** [rts/ghc.mk:325: rts/dist/build/StgCRun.o] Error 1

This also never used to be a problem. @michaelpj - did something change?

@michaelpj - the problem with the line length check is that stylish-haskell actively works against me. For example, consider this (overlong) import in Bitwise.hs:
import Data.Bits (FiniteBits, bit, complement, popCount, rotate, shift, shiftL, xor, zeroBits, (.&.), (.|.))

This is over 100 characters wide. If I reformat it to this (which is within an 80-character width limit):
import Data.Bits (
  FiniteBits, bit, complement, popCount, rotate, shift, shiftL, xor, zeroBits,
  (.&.), (.|.)
  )

When I run fix-stylish-haskell, I get this again:
import Data.Bits (FiniteBits, bit, complement, popCount, rotate, shift, shiftL, xor, zeroBits, (.&.), (.|.))

This is fundamentally a problem with stylish-haskell; not only can it not break overlong lines, it actively works against you if you try. This is one of the (many!) reasons it is a suboptimal code styling tool.
I can certainly ignore the checker as you suggested, but this is a ticking problem that won't go away.

Hi, I am excited to see these builtins in plutus. I wondered what the current status is of the path to active on this CIP? Is this on track for the next HF? I see that @kozross checked all the boxes of the implementation, when is the right time to merge this? I assume that the costing still has to be done?

I think we were waiting for the CIP to make progress. Also this is somewhat reliant on having a keen sponsor pushing it through. If Koz/MLabs have lost interest then someone else would need to take it up.

the sponsor in question is IOG - MLabs performed this work pursuant to IOG funding, we will happily take it up again if IOG deems it worthwhile.

CIP merged, I think the only thing remaining is going through the remaining comments by Michael?

I just realised that integerToByteString is problematic because you can't control the length of the output.
This is important because all operations only work on arguments of equal length.
Padding out a bytestring to the correct length is not zero-cost since we need to create a helper bytestring which we append to the output, resulting in 3 allocated bytestrings rather than 1.
When doing cryptography using these functions, we'd always be mod-ing all arithmetic operations, hence conversion to and from must be fast. In addition, we would never be using negative numbers, instead representing negative numbers as the upper half of the range we choose, hence sign-extension isn't a worry.
If the integer is too big for the chosen length, we cut off the top.
We'd still want to support converting an arbitrary-length integer to bytestring, as there isn't a way of calculating the necessary length effectively. This could be done by regarding a chosen length of 0 as "unlimited". I'm not sure if this is a good idea.
In any case, I assume this should be added to the CIP?

Perhaps the real solution is adding another "extension" operation, one that sign-extends and one that doesn't.

A consequence of the previous scheme is that we can implement an O(1) zero-extension function zeroExtendByteString :: Integer -> BuiltinByteString -> BuiltinByteString that increments the explicit length without touching the underlying array. This might be the only thing we need to add to/change in the specification. Then, if you have x and y, 64-bit signed integers represented as the [0..2^64[ range in Integer, to do an addition, we sum the integers, convert to bytestring (almost no-op), zero-extend to 8 bytes (almost no-op, no-op if already above 8 bytes), AND with 0xFFFFFFFFFFFFFFFF, convert back (almost no-op). This seems reasonable.
An alternative design would be to change slicing such that an out-of-bounds range, rather than being invalid, is filled with zeros.
That would also solve the issue with roughly the same performance (you wouldn't need the AND anymore).
Perhaps including both changes would be wise.
The question is, is the overhead of calling a function low or high relative to the cost of modInteger? If high, it might in the end not be faster than just doing modInteger! Of course, future improvements to the Plutus interpreter might change this.

Once more, apologies for taking so long to deal with this.  However, we've now got some spare capacity and we'd like to make an effort to get this merged in the near future. Is this a good time for you to work on this again?  We can commit to doing some things, like adding costing for the new functions, and maybe adding the new functions to the Agda metatheory and the Plutus Core specification.  FYI, I have a branch where I've updated this to the current state of our main branch and added some of the basic costing functionality.

I'm planning to do a complete review of the code over the next few days, but there are a couple of matters I'd like to raise right  now.  Firstly, I'm thoroughly confused about endianness.  think that all of the functions that involve positions in bytestrings operate the other way round from what's specified in CIP-0058.  Let me spell things out so that we're all on the same page.
Firstly, I'll write the bytestring s given by pack [0x12, 0x34, 0x56] as 0x123456 for brevity (in our concrete syntax that would be (con bytestring #123456)).  Our existing builtins like indexByteString and sliceByteString number the bytes from the left, so takeByteString s 0 would return 0x12 = 18.  The semantics in the CIP writes bytestrings as a sequence of bytes S = s₀s₁...sₖ and S[i] denotes the byte s_i.  The notation for bitstrings is similar.  The semantics for eg testByteString do what you'd expect: given a bitstring S as above, testBitByteString S i is 1 if s_i is set and 0 if it's clear: however, the implementation works from the other end: [(builtin indexByteString) (con bytestring #800000) (con integer 0)] returns False but [(builtin indexByteString) (con bytestring #800000) (con integer 23)] returns True.  The behaviour of writeBitByteString is similar. I think that the reason for this may be that the the operations in Data.ByteString index bytes from the left but those in Data.Bits index bits from the right, so if you're trying to access the bit at position 8i+j in a bytestring of length n you have to adjust one of the positions: either start at the left and look at bit 7-j in byte i, or start at the right and look at bit j in byte n-1-i,  and the latter may be more efficient if you need to look at a lot of bits.  However I think that it'd be better if the implementation does what the specification says, since that would be more consistent with the existing bytestring operations.
The issue of endianness becomes particularly important for conversions between bytestrings and integers.  It isn't explicitly stated in the CIP, but it's clear from the semantics that the intention is that the encoding of integers as bytestrings is supposed to be big-endian, with the leftmost bytes in the bytestring being the most significant hexadecimal digits in the corresponding integer (for example, it says that in S is a bytestring and Z is a bytestring consisting entirely of zero bytes then S and Z·S represent the same integers).  Again though, the implementation uses a little-endian encoding: byteStringToInteger applied to 0x000001 yields 65536, not 1.
However, I think we need to support both big- and little-endian encodings of integers since both occurin practice, and specifically in cryptography.  For example, EdDSA digital signatures use a little-endian encoding for (integers representing) field elements, but RSA , SHA2,  and BLS12-381 all use big-endian encodings.
We could supply two encoding and two decoding functions in Plutus Core, but that might make maintenance a bit tricky.  A simpler solution would be to implement only one encoding (I'd prefer big-endian since that's arguably more intuitive, but that's just my opinion) and also supply a reverseByteString primitive: little-endian encoding could then be expressed as big-endian encoding followed by reversal, and little-endian decoding would be reversal followed by big-endian decoding (or vice-versa); furthermore the extra reversal steps could easily be hidden in higher-level languages targeting UPLC.  A reverseByteString builtin could easily be implemented using Data.ByteString.reverse and would probably be very efficient, so there would be little extra overhead involved in changing endianness.  Whatever we do, it would help to avoid confusion if the name(s) of the relevant builtin(s) mentioned the endiannness, although integerToByteStringLBigEndian would admittedly be unwieldy.

A second issue (as mentioned above) is that many applications use fixed-length encodings of integers (again this happens a lot in cryptography, and the fact that all of the bitwise operations proposed here require both arguments to be of the same length is relevant as well) but the integerToByteString function implemented here just outputs the smallest bytestring that can be used to represent the input, so for example the integer 0 is encoded as the empty bytestring.   It would be possible for users to pad the output to the required width manually, but this would probably be quite expensive.  As mentioned previously, this could either be done by supplying integerToByteString with an extra width argument (but then what do you do if you really don't care about the width) or supplying a built-in padding function.  The latter might be easier, although I have some reservations because it would allow you to create massive bytestrings very quickly, which might be problematic from the point of view of costing.  We might also need "pad on the left" and "pad on the right" functions, although again having reverseByteString might let us get away with only one.

I note also that the CIP proposes both countLeadingZeroesByteString and countTrailingZeroesByteString, but in fact only findFirstSetByteString is implemented.  This should be renamed (or the CIP should be changed), and the other function should also be implemented (although reverseByteString might be helpful here again.

Note that Sundae would benefit dramatically from these bitwise operations; In particular, because transaction inputs get re-ordered, we need to provide an "indexing set" in the redeemer; but because of that, we need to check that each index is distinct, which involves an O(n^2) algorithm, and eats up a huge portion of our execution budget.
With cheap bitwise operators, we could use a bit mask to track indexes we've seen before.

@michaelpj I can provide some psuedocode for what i'm doing now, how many execution units that costs, and some psuedocode for how I was thinking of using bitwise operators, but anything higher fidelity will have to wait until I have some spare time;
Here's what I'm currently doing, more or less:
pub fn get_sorted_orders(
  indexing_set: List<Int>,
  inputs: List<Input>,
) -> List<Output> {
  list.indexed_map(
    indexing_set,
    fn(idx, elem) {
      // Make sure the elements of the indexing set aren't repeated
      // by checking for each element in all the elements leading up to it
      expect !has_in_first_n(indexing_set, elem, idx)
      // Index into this list; in the real implementation I use an
      // unsafe_fast_index_skip that indescriminately skips in large chunks
      // before actually bounds-checking the last few steps
      let input = list.at(inputs, elem)
      input.output
    },
  )
}

In a particular benchmark, the above takes a total of 3,603,290 mem and 1,431,345,727 steps.
If I comment out the expect !has_in_first_n, this drops to 876,338 mem and 336,497,835 steps, saving ~2.8m mem and 1.1b steps;
If I then change it to just loop over the list and return each input (to loosely measure the unsafe_fast_index_skip), it drops to 157,418 mem and 55,464,915 steps, roughly 700k mem and 280m steps.
It's worth noting, I am always bottlenecked on memory, and it's not even close.
Here's what I was thinking I could try with bitwise operators:
pub fn do_get_sorted_orders(
  indexing_set: List<Int>,
  inputs: List<Input>,
  seen_indexes: ByteArray,
) -> List<Output> {
  let [head, ..tail] = indexing_set
  let idx_bit = (1 << head)
  expect seen_indexes && idx_bit == 0
  [list.at(inputs, head), ..do_get_sorted_orders(tail, inputs, seen_indexes || idx_bit)]
}

pub fn get_sorted_orders(
  indexing_set: List<Int>,
  inputs: List<Input>,
) -> List<Output> {
  do_get_sorted_orders(indexing_set, inputs, 0)
}

Or something similar, depending on how the bitwise operators work.  indexing_set is, right now, at most 19 orders, with the hope to push it further,  but likely never hundreds or thousands of entries.

The figures above were for the time taken to evaluate a Plutus Core term applying the builtin to a bytestring.  I ran the benchmarks again and then benchmarked the Haskell function Bitwise.popCountByteString directly,  and I still got a big discontinuity, although in a slightly different place.  The blue dots below are times for the Plutus Core version, the red dots for the Haskell version.

It's similar for integerToByteString.

@kwxm - I have a few suspicions. 8KiB is the page size: this means you're paying an extra penalty on paging, which may explain that discontinuity. The conversion difference is much harder to explain: I'd have to check what I wrote originally.

Ok, thanks.  I'll check my benchmarks again to make sure that I've not made some silly mistake. We haven't seen anything like this for any of the existing builtins, but I'll re-benchmark some of them too in case some change (in the GHC runtime for example) has introduced some new behaviour.

@kwxm - do the benches I wrote for those operations reproduce this behaviour? I haven't run them in a while, but I don't recall seeing such jumps. Maybe I just didn't test with large enough values?

@kwxm - do the benches I wrote for those operations reproduce this behaviour? I haven't run them in a while, but I don't recall seeing such jumps. Maybe I just didn't test with large enough values?

It looks like they do, for popCountByteString at least (I don't think there are any benchmarks involving the eventual implementation of integerToByteString.  I changed the sizes of the inputs to [10,20..1500] and there's a discontinuity for the "block C" implementation: see the graph ("block C" is red, "block unrolled C" is blue; the other two are pretty much straight lines, but they're much steeper so I left them out).

Definitely something I'd use - I use bitwise operators all the time in javascript and other languages. One particular use case that seems relevant - for my recent Smart Life NFT project I am generating Wolfram's rule numbers for 1-d cellular automata by directly converting between 8 individual boolean fields into an 8 bit int which represents the rule number.

I now think the weirdness in the execution time for integerToByteString isn't so bad.  It is a bit strange, but in fact this function is extremely fast (I haven't checked the details, but I think it may not actually have to do much work, perhaps just casting the type of a string of bytes) and a little irregularity in the cost may not be a big problem.  It would be good to understand what's going though in case something changes in the future.

I've added costing code (and a few other things) in this branch and updated the cost model so that it assigns a non-zero cost to the bitwise operations.  We're in the process of transitioning to a new benchmarking machine so the bitwise costs won't be totally consistent with the costs for existing operations; however, it'll at least give us ballpark estimates of costs for programs involving the bitwise operations.

Good to know that this CIP is making progress. I have implemented a pseudo-random number generator that could be a good use case for testing. It's a linear-shift feedback register, and currently, it's prohibitive to run it on-chain, having bit-wise operations would benefit this a lot.

@michaelpj , @kozross , @perturbing
Bitwise operations can be performed on byte strings only in CIP-0058. On the other hand the aritmetic operations can be performed on integers only. In cryptographic algorithms arithmetic and bitwise operations interleave frequently. As a consequence one has to switch between integers and bytestrings via integerToByteString/byteStringToInteger frequently. That conversion may have significant cost and diminish the advantage of using bitwise operations after all. Moreover integerToByteString does not work for negative numbers which requires special handling in cryptographic algorithms, a.k.a. costs you dearly.
I have doubts if bitwise operations on byte strings is an optimal choice for cryptographic applications. Perhaps it's worth revising the design choice by allowing bitwise operations on Plutus integers directly. libgmp which is used as runtime for haskell and Plutus integers has direct support for bitwise operations on integers.

@michaelpj , @kozross , @perturbing Bitwise operations can be performed on byte strings only in CIP-0058. On the other hand the aritmetic operations can be performed on integers only. In cryptographic algorithms arithmetic and bitwise operations interleave frequently. As a consequence one has to switch between integers and bytestrings via integerToByteString/byteStringToInteger frequently. That conversion may have significant cost and diminish the advantage of using bitwise operations after all. Moreover integerToByteString does not work for negative numbers which requires special handling in cryptographic algorithms, a.k.a. costs you dearly.
I have doubts if bitwise operations on byte strings is an optimal choice for cryptographic applications. Perhaps it's worth revising the design choice by allowing bitwise operations on Plutus integers directly. libgmp which is used as runtime for haskell and Plutus integers has direct support for bitwise operations on integers.

I brought up these concerns long ago somewhere and made an alternate design.
It’s better to make the conversion O(1), i.e. merely change the type tag. Wrt. negative numbers, I don’t agree. It shouldn’t be much overhead. You could make another built-in to make it slightly faster though.

This has been superseded by #5654, #5970, and #6090.


Shift description to title/Add new description

Tossing this in to possibly help clarify for people that the countdown is an active edit of a post..

Also added Patreon link. ;)


No kernel messages on tty1 in production

I believe the raspberry pi boards are also affected by this.


Provide an intention to "Execute in Tinker console"

It would be awesome if it was possible to highlight a code snippet in a .php file, press ALT + Enter, and select "Execute in Tinker console".

This is done, will be in the v2.0.0 release!


Implement the "Create App" button

https://github.com/kkebo/try-in-a-playground.swiftpm/blob/72ff7d608435576b7c802998315a7792ecb96fad/Sources/PlaygroundGenerator.swift#L61

I need to implement #5 first.


Support for Universal Purchases in Pilot/Spaceship/Deliver/Fastlane

New Issue Checklist

[X] Updated fastlane to the latest version
[X] I have read the Contribution Guidelines

Issue Description
Currently, managing builds for apps with a Universal Purchase (iOS app and tvOS app using the same bundle ID) causes a number of conflicting results during use.
Setup
Parent folder has two sub-project folders, one for iOS app, one for tvOS app (named AppName and AppNameTVOS). Both share the same bundle ID, and so are both installable once one is purchased. Each project outputs its own independent IPA file, has its own TestFlight management, and has its own screenshots. Each project has its own fastlane directory, with screenshots, meta data. etc.
Sample Fastfile
lane :beta do
    ensure_git_branch(
      branch: 'develop'
      )
    cocoapods
    increment_build_number({
      build_number:  latest_testflight_build_number({
        initial_build_number:  get_build_number(
          xcodeproj: 'TrilbyTVPlayerTVOS.xcodeproj'
        ).to_i
      }) + 1
        })
    gym(
      scheme: "TrilbyTV Player",
      workspace: "TrilbyTVPlayerTVOS.xcworkspace",
      clean: true
    )
    # TODO: snapshot
    sigh
    pilot(
      skip_submission: false,
      distribute_external: false
      )

Issues

latest_testflight_build_number returns build number of iOS app, not tvOS when inside tvOS project folder
In my instance, the latest tvOS testflight build number is 10. It returns 124 based on my iOS app.
agvtool fails to update Info.plist's
I understand this isn't a fastlane issue, just noting it. Incorrect build number passed to agvtool though.

agvtool new-version -all 124
Setting version of project TrilbyTVPlayerTVOS to:
    124.
Also setting CFBundleVersion key (assuming it exists)
Updating CFBundleVersion in Info.plist(s)...
$(SRCROOT)/TrilbyTVPlayerTVOS/TrilbyTVPlayer-Info.plist
Cannot find "$(SRCROOT)/TrilbyTVPlayerTVOS/TrilbyTVPlayer-Info.plist"
$(SRCROOT)/TrilbyTVPlayerTVOS/TrilbyTVPlayerDev-Info.plist
Cannot find "$(SRCROOT)/TrilbyTVPlayerTVOS/TrilbyTVPlayerDev-Info.plist"

This file does exist as its defined in Xcode and app compiles.
Suggestion would be to implement a platform :tvos in the Fastfile to be able to define these settings to the App.
Configuration Files
See above
Environment
fastlane version (run fastlane -v):
1.105.0
Do you use bundler to execute fastlane (i.e. bundle exec fastlane)?
No
Do you use a Ruby environment manager (e.g. chruby, rbenv, rvm)?
rbenv

@dombarnes Thanks for reaching out. We'd love to see a PR to improve performance around TVos. Do other actions face a similar problem or just latest_testflight_build_number. What happens if you have two separate folders completely?
I'm trying to figure out if the issue is local or at the API level. Thanks!

I did some playing with Spaceship and it didn't seem to return any different data where I have a linked tvOS app, only the iOS app data. Not sure if it's API level or not but I'll dig around the code and test with my account and see what I can discover and feed back.
In my mind there could be a 'platform :tvos' definition in the fastfile to set the preference. Or at least separate activity. TvOS apps have slightly different metadata requirements and don't support Watch, iMessage stuff, etc.
I guess that would be up for discussion with the maintainers on best way forward to support this.

@dombarnes Thanks! Can you think of a smaller chunk of work you can take on toward this so we can start working with you on a PR? Hope you are having a great week!

I've started some work on this. See https://github.com/dombarnes/fastlane/tree/universal-purchase
This adds some basic support for a few actions I am able to test with.

@dombarnes Awesome! We'll leave this open 👍

I'd be grateful of a review of https://github.com/fastlane/fastlane/compare/master...dombarnes:universal-purchase
I can't seem to run bundle exec rspec - running from the root just does simplecov, running from fastlane_core returns NameError: uninitialized constant FastlaneCore

It looks like some support for appletvos platform has been added to tools, but isn't part of lib/fastlane/supported_platforms.rb so fastlane fails during validity checking of the Fastfile


[docs] Add Base installation page

Added installation page for Base as a starting point for naming discussion 😂
What should be the name of the product?
the name will appear on the  and product drawer, here are several options:

Base
Base UI
MUI Base

@michaldudak feel free to add more content here or in separate PRs (your call). This PR is intended to be a template for other pages.
Note: please neglect ProductDrawer refinement from this PR

[x] I have followed (at least) the PR section of the contributing guide.

@michaldudak I have changed to Base UI instead. I think it is ready for review.

@michaldudak I have changed to Base UI instead. I think it is ready for review.

I would use MUI Base everywhere and merge. Until we are certain about the name, why using a new one.

Agree - let's use MUI Base for now.

@michaldudak I have changed to Base UI instead. I think it is ready for review.

I would use MUI Base everywhere and merge. Until we are certain about the name, why using a new one?

I might have misunderstood the previous comments, will change to MUI Base

@michaldudak It is changed to MUI Base, do you see anything that might be missing?


Need tests for testing HiveStringType

Have we tested with HiveStringType?
Originally posted by @revans2 in https://github.com/NVIDIA/spark-rapids/pull/1080#r518830502

HiveStringType was removed in Spark 3.1.


Unable to detect changes when a class is moved from one module to another

It has happened several times that a class that is moved from one maven module, to another, for instance
MarfeelGutenberg/MarfeelBoilerpipe/src/main/java/com/marfeel/boilerpipe/html/sax/Tag.java → MarfeelCommonsBoiler/src/main/java/com/marfeel/boilerpipe/html/sax/Tag.java
The plugin does not detect that MarfeelCommonsBoiler, which is used by MarfeelBoilerpipe, should also be taken into account, because now contains the moved class.
It just builds MarfeelBoilerpipe and its dependant modules, and because of that the build fails

Hi,
I think that is a bug in DifferentFilesNative.java, can you test the same case with useNativeGit=false and see if it still doesn't include MarfeelCommonsBoiler ?
Thanks,

Yeah, you are right I am using useNativeGit=true
I'll check it in a while and let you know, thanks for the fast feedback!

Sorry for the late feedback. Works fine with useNativeGit=false


Added standalone example app.

Hey there,
The example inside the examples folder was just a file, i've added a full application project inside an appropriately named sub folder ( Notepad ) as an example, so that the user only needs to do npm install and then do npm start to get an example application up and running. Hope that adds value.
Appreciate the great work on the library once again!. :) 👍

Thanks it looks good.


I think, There are some errors in the documention

[Enter feedback here]
section:

userRepositoryRepository | (Required) Select the name of GitHub repository in which GitHub releases will be created.

should be something more like
Select the name of GitHub repository from which GitHub releases will be downloaded.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8cebdaea-9b1f-e65a-a789-6571b5e28c03
Version Independent ID: ca0de77d-d8a7-d766-90e6-929bdb2c877f
Content: Download GitHub Release task - Azure Pipelines
Content Source: docs/pipelines/tasks/utility/download-github-release.md
Product: devops
Technology: devops-cicd-tasks
GitHub Login: @ashokirla
Microsoft Alias: ashkir

@Raffaello Thank you for the feedback, please consider becoming a contributor.
The documentation has been updated and is pending review.


Fixing Creation of Opensuse data Files

As mentioned in #145 issue. All the OpenSUSE data files that are being generated have an empty JSON file, that was due to a buggy code that Arsh and I tried to correct. We use to have separate chunk of code for creation of different files, In this PR I've use nested for loop and try to reduce down the repetition and the lines of code.
Once this get a green flag, I'll subsequently update the data files in the data repository that are empty right now.
@pleia2 and @arshPratap please take a look and let me know if there's anything that needs to be done.

Thanks @Princee215 ..works really well..just some minor fixes thats all

@Princee215 lemme know when you have fixed the changes I have suggested ..  I would love to approve this PR
@pleia2 would love to hear your thoughts on this .
Thanks

Sure, thanks @arshPratap, i see these changes. And I'll try to edit as soon as possible.

Great Work @Princee215


for.. in.. , sublist always get 0.

hi guys ,i'm using RealmSwift 0.92
i have a class
class MediaSentenceWord: Object {
   let librarywords = List<LibraryWord>()
}

var results = realm.objects(MediaSentenceWord)
for i in 0...results.count-1{
       println(libResults[i].librarywords.count) // every thing ok
}

but if i write this ,there is a problem:
for item in results{
        println(item.librarywords.count) // always 0  !!!!!
}

We can confirm that this is an issue with Realm Swift preventing relationships from properly resolving during Swift's fast enumeration. We'll have a fix out shortly. Thanks for reporting this.


Hotfix within adafruit_wm8960.WM8960.volume setter

I somehow let slip a blatant error within the helper class which prevents DAC output from working. This small update fixes the error.

Thanks for the quick merge, @dhalbert!


gnu(8928)=6497

This took 6628.908 seconds to compute.

gnu(8928)=6497
GAP      4.8.3
GrpConst 2.5
Runtime: 4831051 ms
Isomorphic groups eliminated
Validated on Linux server (AMD Opteron 6376 1400 MHz processor, 512 GB RAM)


update ak receiver transforms

This PR updates the AK receiver transforms to conform to their required specs
If you are suggesting a fix for a currently exploitable issue, please disclose the issue to the prime-reportstream team directly outside of GitHub instead of filing a PR, so we may immediately patch the affected systems before a disclosure. See SECURITY.md/Reporting a Vulnerability for more information.
Test Steps:

run ./prime test
run ./gradlew testIntegration

Changes

*re-ordered elements to appear in the order they appear in outbound message for readability
*Added default value for MSH-5-1
*Added valueset for FIPS county codes
*Added valueset for patient ethnicity
*Updated Integration Tests
*Updated receiver settings in folder

Checklist
Testing

[x ] Tested locally?
[x ] Ran ./prime test or ./gradlew testSmoke against local Docker ReportStream container?
[ ] (For Changes to /frontend-react/...) Ran npm run lint:write?
[ ] Added tests?

@victor-chaparro The AOE date time issue hasn't been resolved but I wanted to get all the other stuff out of the way so we could get all the testing except for that one issue out of the way while we still work on it.


fd: 7.0.0 -> 7.1.0

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

Success on x86_64-linux (full log)
Attempted: fd
Partial log (click to expand)
post-installation fixup
moving /nix/store/s47aa5bqij139zdzsh6ad156l2a3fnls-fd-7.1.0/man to /nix/store/s47aa5bqij139zdzsh6ad156l2a3fnls-fd-7.1.0/share/man
shrinking RPATHs of ELF executables and libraries in /nix/store/s47aa5bqij139zdzsh6ad156l2a3fnls-fd-7.1.0
shrinking /nix/store/s47aa5bqij139zdzsh6ad156l2a3fnls-fd-7.1.0/bin/fd
gzipping man pages under /nix/store/s47aa5bqij139zdzsh6ad156l2a3fnls-fd-7.1.0/share/man/
strip is /nix/store/553rihc190vsyy8b22iqcq25a6489h8y-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/s47aa5bqij139zdzsh6ad156l2a3fnls-fd-7.1.0/bin
patching script interpreter paths in /nix/store/s47aa5bqij139zdzsh6ad156l2a3fnls-fd-7.1.0
checking for references to /build in /nix/store/s47aa5bqij139zdzsh6ad156l2a3fnls-fd-7.1.0...
/nix/store/s47aa5bqij139zdzsh6ad156l2a3fnls-fd-7.1.0

Success on aarch64-linux (full log)
Attempted: fd
Partial log (click to expand)
post-installation fixup
moving /nix/store/ljhpcf16g3jyjziyv1mj95n7x0cqd15m-fd-7.1.0/man to /nix/store/ljhpcf16g3jyjziyv1mj95n7x0cqd15m-fd-7.1.0/share/man
shrinking RPATHs of ELF executables and libraries in /nix/store/ljhpcf16g3jyjziyv1mj95n7x0cqd15m-fd-7.1.0
shrinking /nix/store/ljhpcf16g3jyjziyv1mj95n7x0cqd15m-fd-7.1.0/bin/fd
gzipping man pages under /nix/store/ljhpcf16g3jyjziyv1mj95n7x0cqd15m-fd-7.1.0/share/man/
strip is /nix/store/4md2i310zklkkl5j41yw70gcwgn4kav5-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/ljhpcf16g3jyjziyv1mj95n7x0cqd15m-fd-7.1.0/bin
patching script interpreter paths in /nix/store/ljhpcf16g3jyjziyv1mj95n7x0cqd15m-fd-7.1.0
checking for references to /build in /nix/store/ljhpcf16g3jyjziyv1mj95n7x0cqd15m-fd-7.1.0...
/nix/store/ljhpcf16g3jyjziyv1mj95n7x0cqd15m-fd-7.1.0


Flashing fails with this compiled version while 1.7 is doing fine.

1.7 supplied with Arduino 1.8.9 works just fine, but when I compile and run this version here it fails as below.
The command given is like this:
bossac --port=ttyACM0 -i -U -e -w -R -v .bin
and the relevant part of the reply:
Done in 0.740 seconds
Write 49716 bytes to flash (777 pages)
[==============================] 100% (777/777 pages)
Done in 0.255 seconds
Verify 49716 bytes of flash
[==============================] 100% (777/777 pages)
Verify failed
Page errors: 128
Byte errors: 7956

I have the same issue I think...
I've compiled versions 1.7.0, 1.8 and 1.9.1 on a Raspberry Pi 3B+ which I use to program a SparkFun SAMD21 Mini Breakout board, (ie. an Arduino Zero clone).
Both 1.8 and 1.9.1 fail in the same way, (see below), whereas 1.7.0 works? One thing I've noticed is that 1.7.0 successfully detects the USB port and connects at the higher speed, (Connected at 921600 baud), whereas the later versions don't:

sudo bossac -i -d --port=ttyACM0 --usb-port=true -e -w -v -R '/home/programmer/Sampler_Controller.bin'
Send auto-baud
Set binary mode
version()=v2.0 [Arduino:XYZ] Dec 20 2016 15:36:39
Connected at 115200 baud
readWord(addr=0)=0x20007ffc
readWord(addr=0xe000ed00)=0x410cc601
readWord(addr=0x41002018)=0x10010305
write(addr=0x20004000,size=0x34)
SAM-BA operation failed

I also find that I have to run the following command beforehand to ensure that bossac will work. (The SAMD21 board also occasionally needs to be unplugged and re-inserted).
sudo stty -F /dev/ttyACM0 speed 1200 cs8 -cstopb -parenb

If I don't do this I get the following error:

Set binary mode
Send auto-baud
Set binary mode
No device found on ttyACM0

Did some bisecting and got this commit causing issues:
commit ee5a651aacaf2c3fbcd539789574618a425d1a06 (refs/bisect/bad)
Author: Scott Shumate<EMAIL_ADDRESS>Date:   Sun May 21 09:58:01 2017 -0500
Refactored to encapsulate all device-specific knowledge into the Device
class including flash creation to prevent Samba from being a dumping
ground for miscellaneous device features.

@RatGit
the semantic of -U parameter has been changed from 1.7.
You may want to try to replace --usb-port=true or -U true to just -U

@pjalocha @RatGit
try also to add -o 0x2000 to the command line (I suppose you're running a SAMD21 based board), previously the offset to skip the bootloader size was built-in in bossac, now you must specify it via command line.

@RatGit re: the board needing to be unplugged and then plugged back in, I see the same due to a race between the 'stty' call, the board detaching, and Bossa attaching.  If the board hasn't finished restarting then Linux puts the board on ttyACM1 instead of ttyACM0

I confirm zephyr's version is working fine on arduino_nano_33_ble
I can provide some traces if it helps
/usr/local/bin/bossac-1.9.1-14-g3532de8  --version
Basic Open Source SAM-BA Application (BOSSA) Version 1.9.1-14-g3532de8

/usr/local/bin/bossac-1.9.1-17-g89f3556 --version
Basic Open Source SAM-BA Application (BOSSA) Version 1.9.1-17-g89f3556


Support partial words in search query

Please see https://github.com/dotnet/cli/issues/9128#issuecomment-384004549 for more details

Responded on the original issue https://github.com/dotnet/cli/issues/9128#issuecomment-384004549
The solution here is to use the autocomplete API and not the search API.


ENH try osx

Checklist

[x] Used a fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin rerender

@conda-forge-admin rerender

@chrisburr The osx build of the python extension failed with a symbol not found (_lua_strlen). Do you know how to fix this?
I commented out the python extension build on osx for now.

No idea I was hoping you might 😄
Regardless I think it's fine to merge with them disabled.

I assume this has to do with getting liblua.a linked into the python extension when it is built. We could try a patch at another time.

ok @chrisburr this one is ready for review!

@conda-forge-admin, please rerender

Thanks @beckermr!


KinematicBody2D: odd move_and_slide behaviour when exactly on tile edge

Godot version:
3.2.3-official
OS/device including version:
Ubuntu Linux 20.04
Issue description:
Possibly related to #33833.
When using KinematicBody2D's move_and_slide or move_and_slide_with_snap to enact gravity, I've noticed that when a body is exactly on the diagonal edge of a tile:

is_on_floor() returns false
is_on_wall() returns true
The body can't travel any further, but keeps gaining velocity (when setting it from the result of move_and_slide)

So we essentially end up with a Mario 64 speedrunner's dream:

Interestingly, the value of y tends to fluctuate, and sometimes (seemingly non-deterministically) the body unhooks itself and so goes flying downwards.
Steps to reproduce:

Position a KinematicBody2D on the edge of a tile
Apply gravity to it
Note the weird behaviour of its position and velocity

Minimal reproduction project:
floor-wall-test.zip

Related to #43012 and #43014.

Related to #43012 and #43014.

I've reproduced this in 3.2.4-rc1.

I've reproduced this in 3.2.4-rc1.

I can confirm that the issue is still reproducible in v3.5.3.stable.official.
Here's a workaround I used:
    if is_on_wall() && is_equal_approx(position.x, round(position.x)):
        position.x += 0.001


fix createPostageBatch function arguments in the README & testcases

createPostageBatch function arguments in the README & testcases need to be fixed / updated with realistic values that can be used as reference

What is realistic value as reference? I am thinking about a week for expiry, with depth=24 (44GB) could work.

According to this page: https://www.ethswarm.org/get-bzz
The current price 0.3331 BZZ for 1GB per month. So with the proposed values it would cost (44 * 0.3331 / 4) = 3.6641 BZZ, that is at today's price is about 1.47 USD, which I think is reasonable for testing.

What is realistic value as reference? I am thinking about a week for expiry, with depth=24 (44GB) could work.

Depth 20 would be sufficient.
But the amounts are too low for the stamp  purchase to succeed.
The amount has to have a TTL of more than a day it seems. So approximately any amount > 500000000

The actual minimum amount is affected by the storage price when the stamp batch is purchased.   So if the price continues to go up, the amount will need to track.  Therefore, it is tricky to document a "minimum" that "works" when the future is unknown.

Maybe we need a new /stamps API that provides the actual current minimums given the current swarm conditions?

True. Valid point!