x/tools/cmd/callgraph: calls from unreachable unexported methods not reported in callgraph

Go version
go version go1.22.0 darwin/amd64
Output of go env in your module/workspace:
GO111MODULE=''
GOARCH='amd64'
GOBIN='/Users/aggnolek/go/bin'
GOCACHE='/Users/aggnolek/Library/Caches/go-build'
GOENV='/Users/aggnolek/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/aggnolek/go/pkg/mod'
GONOPROXY='*'
GONOSUMDB='*'
GOOS='darwin'
GOPATH='/Users/aggnolek/go'
GOPRIVATE='*'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/aggnolek/sdk/go1.22.0'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/aggnolek/sdk/go1.22.0/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/aggnolek/gorepos/aggnolek/scratchpad/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/x7/2f8ynt3954s4y78yt_54v9fr0000gs/T/go-build886972198=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?
go.mod
module scratchpad

go 1.21

example2/main.go
package main

func called() {}

type unexported struct{}

func (u unexported) Func1() {
	called()
}

type Exported struct{}

func (e Exported) Func1() {
	called()
}

func main() {

}

Ran callgraph -algo={algo} ./example2 for algo in [static, cha, rta, vta]
What did you see happen?
callgraph -algo=static ./example2
(scratchpad/example2.Exported).Func1	--static-14:8-->	scratchpad/example2.called

callgraph -algo=cha ./example2
(scratchpad/example2.Exported).Func1	--static-14:8-->	scratchpad/example2.called

callgraph -algo=rta ./example2
{Empty Output}

callgraph -algo=vta ./example2
(scratchpad/example2.Exported).Func1	--static-14:8-->	scratchpad/example2.called

What did you expect to see?
callgraph -algo=static ./example2
(scratchpad/example2.Exported).Func1	--static-14:8-->	scratchpad/example2.called
(scratchpad/example2.unexported).Func1	--static-8:8-->	scratchpad/example2.called

callgraph -algo=cha ./example2
(scratchpad/example2.Exported).Func1	--static-14:8-->	scratchpad/example2.called
(scratchpad/example2.unexported).Func1	--static-8:8-->	scratchpad/example2.called

callgraph -algo=rta ./example2
{Empty Output}

This is because rta only includes reachable funcs by design
callgraph -algo=vta ./example2
(scratchpad/example2.Exported).Func1	--static-14:8-->	scratchpad/example2.called
(scratchpad/example2.unexported).Func1	--static-8:8-->	scratchpad/example2.called

Note that the link from example2.unexported).Func1 to called is present when example2.unexported).Func1 is forced to be reachable. For example:
example2/main.go
func main() {
	unexported{}.Func1()
}

callgraph -algo=static ./example2
(scratchpad/example2.unexported).Func1	--static-8:8-->	scratchpad/example2.called
(scratchpad/example2.Exported).Func1	--static-14:8-->	scratchpad/example2.called
scratchpad/example2.main	--static-18:20-->	(scratchpad/example2.unexported).Func1

callgraph -algo=cha ./example2
(scratchpad/example2.unexported).Func1	--static-8:8-->	scratchpad/example2.called
(scratchpad/example2.Exported).Func1	--static-14:8-->	scratchpad/example2.called
scratchpad/example2.main	--static-18:20-->	(scratchpad/example2.unexported).Func1

callgraph -algo=rta ./example2
(scratchpad/example2.unexported).Func1	--static-8:8-->	scratchpad/example2.called
scratchpad/example2.main	--static-18:20-->	(scratchpad/example2.unexported).Func1

This is because rta only includes reachable funcs by design
callgraph -algo=vta ./example2
(scratchpad/example2.unexported).Func1	--static-8:8-->	scratchpad/example2.called
(scratchpad/example2.Exported).Func1	--static-14:8-->	scratchpad/example2.called
scratchpad/example2.main	--static-18:20-->	(scratchpad/example2.unexported).Func1

This likely has to do with the same usage of ssautil.AllFunctions (at least for vta) that resulted in this bug for the deadcode command.

@adonovan Given your closeness to the likely similar prior issue

Agree with respect to rta, although based on my understanding both static and cha will require separate fixes as those don't take in sourceFuncs.

Based on my early testing, I believe you'll also need to filter out nil results for fn, as prog.FuncValue returns nil for interface methods and those nils cause panics within vta.CallGraph

Agree with respect to rta, although based on my understanding both static and cha will require separate fixes as those don't take in sourceFuncs.

The bug in -algo=static was fixed yesterday by https://go.dev/cl/609280.
The fix for the bug in CHA (which is really a bug in AllFunctions) is something along the lines of https://go.dev/cl/609281, but as it stands that CL would cause CHA's analysis of main packages to no longer treat exported symbols as entry points, which is sound, but I need to think a little more about whether this is consistent with the expectations we have set for CHA. For example, that CL would cause the -algo=cha output on the example above to report the empty set (just like RTA), which is sound but is counter to the expectations you had when filing this bug report. I am inclined to think that the problem of executing the CHA analysis is really orthogonal to the problem of how one gathers up the initial set of functions it uses (and which of those functions are considered address-taken). We may need new API to do it justice.

On further reflection, I think the callgraph tool is working as intended when it shows no call edge from unexported.Func1 to called in your first example, because the unexported type is unreachable according to the reachability algorithm implemented by AllFunctions. AllFunctions has certainly had a history of bugs (see #69291), but it has always pretended to use a linker-style reachability algorithm, and any such algorithm should report that unexported is not reachable.
If you were to use a different algorithm to gather up functions ("ReallyAllFunctions(prog)"), the CHA algorithm would analyze the dynamic calls within even the unreachable functions. (We don't have a test for that because both cmd/callgraph and the CHA tests both use AllFunction, but I'm confident it would work.) So if you really need that behavior, that's something you could do.
My apologies for the various contradictory things I have said on this and other related callgraph issues this week.


PSD support

Forking from https://github.com/nwtn/grunt-respimg/issues/16#issuecomment-119113979

Is it possible to throw PSD files at this to convert to JPG and run through optimisation?

Good idea. I'll keep it in mind for a future update.

closing because this repo is not maintained


Release Notes opened in VSCode keeps returning to the top of the page after change focus back

VSCode Version: 1.45.0
OS Version: Windows 10

the issue's existed since many versions ago.
Steps to Reproduce:

open Release Note from menu Help->Release Notes, then scroll down to not see the top the page
switch to another tab in editor
switch back to Release Notes
now it returns back to the top the page of Release Notes.

Does this issue occur when all extensions are disabled?: Yes

/duplicate of #94945


Additional spaces in version 4.x?

Hi,
I am seeing some additional spaces after and before [ and ] as well as around # characters in version 4.0.2.
Is that something I can configure to avoid?
version 2.3.4:
SELECT
  [J7].[ProductGroupID] AS [###2540],
  [J7].[ProductGroupName] AS [###2541],
  [J7].[ProductID] AS [###8331],
  [J7].[ProductName] AS [###8332],
  DAI_.[Company] AS [###2544],
  COUNT(*) AS AGGREGATION_COUNT,
  SUM(DAI_.[Amount]) AS [###2539_SUM]

version 4.0.2
SELECT
  [ J7 ].[ ProductGroupID ] AS [ # # # 2540 ],
  [ J7 ].[ ProductGroupName ] AS [ # # # 2541 ],
  [ J7 ].[ ProductID ] AS [ # # # 8331 ],
  [ J7 ].[ ProductName ] AS [ # # # 8332 ],
  DAI_.[ Company ] AS [ # # # 2544 ],
  COUNT(*) AS AGGREGATION_COUNT,
  SUM(DAI_.[ Amount ]) AS [ # # # 2539_SUM ]

Thanks!

I suppose you're using the default standard SQL formatter. I suggest you switch to more specific Transact-SQL:
format("SELECT * ...", { language: "tsql" });


Remove backwards compatible fields

Don't want to merge this yet. Want to make sure all boards are doing COALESCE(old, new) or some sort of CASE WHEN on measure_name for these fields. Would be nice if you could update records in timestream but nope...
Dropping these cause it's just storing data we are duplicating, hence increasing costs.

I'll pop an announcement about this in the test reporting channel as well before merging.

I'll merge this in a bit. Still double checking dash boards for usage.

I'll pop an announcement about this in the test reporting channel as well before merging.

Announcement.
https://d2l.slack.com/archives/C05MMC7H7EK/p1732808334633299?thread_ts=1731624162.047039&cid=C05MMC7H7EK


Add more methods

Add more fingerprinting methods:
cpu
physical memory
gpu
mac address
ip address
See the tests for this branch run on ubuntu-latest, windows-latest, macos-latest:
https://github.com/open-runtime/runtime_digital_fingerprint/actions/runs/10779963878

Suggestions / comments welcome


install feature is always on, add install-to-disk

I consider bootc install to-filesystem support a key feature of bootc. In theory today one can still set up a system directly with ostree and we will continue to support that.
But things like logically bound images we do want to be initialized from the start and that's with bootc install to-filesystem.
Maintaining the feature just has a logistical annoyance any time one touches the install code as we often end up needing to carefully #[cfg(feature = "install")] in many places in an infectious way.
Also as we head towards enabling factory reset
https://github.com/containers/bootc/issues/404
we really do want some of the install code enabled there.
However, to-disk is much more of a "demo". I don't want bootc to grow too much knowledge around block devices. Complex setups (LVM, LUKS) etc. are the domain of external installers and provisioning tools.
So the feature gate is now on that (which is still on by default).
We ended up with more #[cfg(feature = "install-to-disk")] than I'd have liked, but some of that can be fixed subsequently.

Rebased :surfer: for trivial conflict with https://github.com/containers/bootc/pull/990

/packit retest-failed


v7 and breaking changes

Hi,
First of all congrats for the great package.
Reading the v7 changelog saw that Both child and children have moved into the args for all widgets.
That is an ultimate stopper for us as we have lots and lots custom json templates stored in our backend.
Is it something you can consider not doing it?
Thanks

Unfortunately, to meet the goals of the 7.0.0 release they need to move because they aren't technically special.  When all widgets were "built by humans" we could make the distinction regarding where those attributes go.  But once they are built by a code generator, the code generator just looks at all the attributes and groups them.  It doesn't make sense to try to move some to the args but not others, and the problem gets worse when longer term maintainability is considered.
Having child and children outside the args had other unfortunate impacts.  For example "builder".  Should that be outside or inside the "args"?  It's "people-y" the same as child / children.  Issues like the "body" on a Scaffold (rather than it being a "child") also led to some awkward compromises.  That led to the decision that if it's an attribute of the widget, it goes to "args" and if it's an attribute to the framework (type, id, listen, etc) it can live outside.
A few notes of mitigation...

The next release is a major release with announced breaking changes.  You have an option to not upgrading and sticking with 6.0.0.  Assuming your pubspec.yaml uses the standard ^6.0.0 (or whatever)
Realizing that this was a big deal, the 7.0.0 branch also includes a migration script that can automatically migrate existing JSON templates to the new structure for all known breaking changes.  You can apply that same script to your custom templates.  That script was what was used to migrate all the example templates.

Thanks @jpeiffer for your answer probably will stick with v6


Stuck at requesting vehicle info

Hello,
I just installed the latest version and the job is stuck at requesting vehicle info for the past 5 minutes. In MQTT, I see there's a new entry but this entry says it's available = false.
I'm from Canada and I think we have a different onstar url, could that be the problem?

Ok, I restarted the job and now I'm getting this error


No interface of certain items/blocks

Describe the Bug
Right-clicking on filter, attribute filter, train schedule, schematicannon, schematic table, toolbox and crafting table doesn't do anything.
I've tested this in single-player world and everything was working fine, but on my Fabric Server there is this bug. Nothing pops on in console as well, there isn't any crashes or error reports.
I've recorded video that at least shows how it looks
Reproduction Steps

Join server with modpack
Try to use any item or block that I've listed

Expected Result
Poped up ui on right-clicking item/block
Screenshots and Videos
https://youtu.be/n0y1uVxqMmE
Crash Report or Log
No response
Operating System
Windows 11
Mod Version
0.5.0i
Minecraft Version
1.19.2
Other Mods
Full list of Server-side mods:
1.19.1-2-Origins-1.7.1 (1).jar
1.19.2-Fabric-API-0.76.0-1.19.2.jar
1.19.2-FabricTailor-2.0.1 (1).jar
1.19.2-YUNG-s-Better-Desert-Temples-v2.2.2-Fabric.jar
1.19.2-YUNG-s-Better-Mineshafts-v3.2.0-Fabric.jar
1.19.2-YUNG-s-Better-Nether-Fortresses-v1.0.5-Fabric.jar
Amplified_Nether_1.19.3_v1.2.1.jar
Another-Furniture-2.1.2-Fabric-1.19.2.jar
areas-1.19.2-4.3.jar
arrp-0.6.4.jar
artifacts-7.1.1+fabric.jar
bettercombat-fabric-1.7.1-1.19.jar
capybara-1.19-2.0.1.jar
chipped-fabric-1.19.2-2.1.3.jar
Chunky-1.3.38.jar
cloth-config-8.2.88-fabric.jar
collective-1.19.2-6.50.jar
ColorfulAzaleas-fabric-1.19.2-2.3.1.jar
CorgiLib-fabric-1.19.2-<IP_ADDRESS>.jar
Couplings-1.9.1-1.19.jar
create-fabric-0.5.0.i-1003+1.19.2.jar
crittersandcompanions-fabric-1.19.2-2.0.1.jar
ctm-fabric-1.0.0+1.19.jar
ctov-3.2.3-1.19.4.jar
drinkbeer-refill-1.19.2-1.0.1.jar
easyauth-mc1.19.1-3.0.0-14.jar
emoji-type-2.0.1-1.19.jar
emotecraft-for-MC1.19.2-2.2.7-b.build.47-fabric.jar
Fabric-Creeper-Overhaul-2.0.8-1.19.2.jar
fabric-language-kotlin-1.8.4+kotlin.1.7.20.jar
FarmersDelight-1.3.9-1.19.X.jar
Female-Gender-Mod-3.0.1-1.19.2.jar
geckolib-fabric-1.19-3.1.40.jar
handcrafted-fabric-1.19.2-2.0.6.jar
immersive_paintings-0.6.0-1.19.2-fabric.jar
itemblacklist-1.19.2-1.2.4.jar
jei-1.19.2-fabric-<IP_ADDRESS>3.jar
ledger-1.2.5.jar
mcw-trapdoors-1.1.0-mc1.19.2fabric.jar
medievalweapons-1.4.4-1.19.2.jar
minecells-1.4.3.jar
Mod-Menu-v4.2.0-beta.2-for-1.19.2.jar
moonlight-1.19.2-2.2.38-fabric.jar
naturalist-fabric-3.0.3a-1.19.2.jar
NethersDelight-1.0.1v-1.19.2.jar
Nullscape_1.19.3_v1.2.1.jar
originsumbrellas-1.5.4.jar
Pehkui-3.4.0+1.14.4-1.19.2 (1).jar
player-animation-lib-fabric-1.0.2-1.19.2.jar
resourcefullib-fabric-1.19.2-1.1.23.jar
riding_partners-1.0.0-fabric.jar
Rotten-Creatures-Fabric-1.19.2-v1.0.1.jar
simplechat-1.6.1.jar
supplementaries-1.19.2-2.3.17-fabric.jar
tl_skin_cape_fabric_1.19.1_1.19.4-1.30.jar
trinkets-3.4.2.jar
vcinteraction-1.19.2-1.0.3.jar
voicechat-fabric-1.19.2-2.4.9.jar
WondrousWilds-FABRIC-1.19.2-1.1.6.jar
YungsApi-1.19.2-Fabric-3.8.8.jar
YungsBetterOceanMonuments-1.19.2-Fabric-2.1.0.jar
Additional Context
No response

Are all client side mods also installed on the server?
Can you check tps while trying to open the gui to see if the server is not frozen? You can do this by installing spark on both server and client and running the commmand /spark tps
https://modrinth.com/mod/spark

Just maked sure that mods in server-side and client-side folders compare to each other. And spark mod shows that there isn't any tps lowering or other freezes. Everything aside this items works exactly fine

Also tried to delete all config files and generate them all over again to be sure

need the server log

closing for inactivity


Update Obsolete check to bulk check all specs in one statement vs one statement per spec.

Pushed to development branch


Pagination

We are getting all the images to find out the number of pages , insted of showing all the pages show only say next 5 pages , so only count * 5 images to be loaded to find length and not all

Right now number of items per page is 5 , on uploading 5 images only one page must be shown but it instead shows 2 page , 2nd page with no image ....


CARLA: send frames over visionipc

@deanlee this is mostly being held up because the YUV -> RGB conversion is slow. want to write a GL shader so we can ditch the RGB buffers?

Do we still need to use EGLImageTexture?

I have started work on this, as part of cleaning up the Carla simulation. Got the UI video working via vipc located in the bridge. I discovered "the hard way" that bridge.py needs to produce both RGB and YUV video, to satisfy both the user interface and modeld. No problem, lots of learning along the way. I am wondering, however, why RGB <-> YUV conversions aren't centralized in the vipc server. Server could listen to either, but produce both formats. I will continue to focus on adding the conversion to YUV into the bridge. Do you think openCL (as in existing camera code) is the right way to convert, or perhaps something else? I'm doing my own investigation of conversion methods too.

I believe @nuwandavek is writing a small hack to get this merged in during the hackathon, but it seems like you're interested in doing this properly. Ultimately, we want to remove the RGB stream. The only thing left to do here is make the UI work with the YUV stream, which should be relatively straightforward. Everything else that uses visionipc already uses the YUV streams.

Thanks for the guidance. Moving toward a single protocol makes a lot of sense. I'll focus on producing YUV from the bridge.

Looking into ways to do the conversion, I came across pyopencl. Seems like this would allow me to port the existing rgb_to_yuv conversion routine (written in C++/opencl ) over to the python environment of bridge.py without too much fuss. It looks like it fits well with numpy too. Does this approach make sense to you?

I have modified bridge.py to do the conversion to YUV, using pyopencl to call the existing rgb_to_yuv.cl kernel. Seems to run,  invoking the kernel and moving buffers around, but it needs testing. I tried the draft ui mods for using YUV, but got a green screen, and as I recall that is in line with some troubles that were reported for that mod. Is a working version of the YUV UI screen expected soon? I could try fixing it myself, but that would probably be a duplicated effort, with a whole new learning curve.

#22905 can display YUV correctly.  you can run replay --demo --yuv to see the result .have you checked your conversion in other ways?

Thanks, knowing the display is good helps a lot. I'll have to look elsewhere for the problems.


Error in the Chebyshev reconstructor

In the reconstruction of the rod, I am using the DDCT as from the Noegue paper. However I am using a different set of Chebyshev points so the matrix has to be adapted to our points (the extrema one) as explained in Wikipedia

solved now there's a class for that


[2] Press "u" to update snapshots from watch mode

Pressing "u" causes all failing snapshots to be updated. There is no way to select individual snapshots to update yet, but that's something we could think about for the future.
Note: this diff is based off of the changes for #1475, I'm not sure if there's a way to do stacked PRs.

Current coverage is 62.83% (diff: 2.17%)

Merging #1479 into master will decrease coverage by 0.47%

@@             master      #1479   diff @@
==========================================
  Files            96         96
  Lines          3608       3635    +27
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           2284       2284
- Misses         1324       1351    +27
  Partials          0          0

Powered by Codecov. Last update 081bdd5...2e30570

This is super awesome. Would you mind rebasing it so I can merge it?
Eventually we should move the watcher stuff into a separate file, but for now this is fantastic.

See #1508.


Structs missing in generated go code

Description
Some simple types are not generated under given circumstances. The original XSD is not created by me, but is i understand, is old and widely used.

Steps to reproduce the issue:

See reproducible minimal example code below.
xgen -l Go -p internal -i resources/xsd/ -o internal/model/
Check the output.

Describe the results you received:
AAA has a struct in go code created.
Describe the results you expected:
GGG and DDD has types in go code created as well.
 This seems to be the issue, CCC is not generated with this restriction enabled -->
                        <xsd:minLength value = "4" fixed = "true" />
                    </xsd:restriction>
                </xsd:simpleType>
            </xsd:element>
        </xsd:sequence>
    </xsd:complexType>

    <xsd:simpleType name="DDD">
        <xsd:annotation><xsd:documentation>EEE</xsd:documentation></xsd:annotation>
        <xsd:restriction base="ak:GGG">
            <!-- While CCC is ok, this is okay as well. While CCC has the invalid restriction: if any of the following it commented in, DDD is not generated anymore as well. -->
<!--            <xsd:whiteSpace value="collapse" />
<!--            <xsd:minLength value="1" />-->
<!--            <xsd:maxLength value="43" />-->
        </xsd:restriction>
    </xsd:simpleType>

    <xsd:simpleType name="GGG">
        <xsd:annotation><xsd:documentation>FFF</xsd:documentation></xsd:annotation>
        <xsd:restriction base="xsd:string"/>
    </xsd:simpleType>

</xsd:schema>

Environment details (OS, physical, etc.):

Is this the same issue?
<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema
        xmlns:xsd="http://www.w3.org/2001/XMLSchema"
        targetNamespace="http://example.com/schemas"
        xmlns:ak="http://example.com/schemas"
        elementFormDefault="qualified">

    <xsd:simpleType name="CCC">
        <xsd:restriction base="ak:GGG">
            <xsd:minLength value = "4" fixed = "true" />
        </xsd:restriction>
    </xsd:simpleType>
    <!-- This seems wrong, it should be GGG, not string -->
    <!--    // CCC ...-->
    <!--    type CCC string-->

    <xsd:simpleType name="GGG">
        <xsd:annotation><xsd:documentation>FFF</xsd:documentation></xsd:annotation>
        <xsd:restriction base="xsd:string"/>
    </xsd:simpleType>
    <!--    // GGG is FFF-->
    <!--    type GGG string-->

</xsd:schema>

Or this?
<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema
        xmlns:xsd="http://www.w3.org/2001/XMLSchema"
        targetNamespace="http://example.com/schemas"
        xmlns:ak="http://example.com/schemas"
        elementFormDefault="qualified">

    <xsd:element name="CCC">
        <xsd:simpleType>
            <xsd:restriction base="ak:GGG">
            </xsd:restriction>
        </xsd:simpleType>
        <!-- This seems wrong, it should be GGG, not string -->
        <!--        // AkGGG ...-->
        <!--        type AkGGG string-->
    </xsd:element>
    <!-- This seems very wrong, it should be a struct -->
    <!--    // CCC is FFF-->
    <!--    type CCC *CCC-->

    <xsd:simpleType name="GGG">
        <xsd:annotation><xsd:documentation>FFF</xsd:documentation></xsd:annotation>
        <xsd:restriction base="xsd:string"/>
    </xsd:simpleType>
    <!--    // GGG ...-->
    <!--    type GGG string-->

</xsd:schema>

I have the same issue. Simple Types are not being created

While trying to comprehend what is the cause, I think there is an issue when we find a xsd:simpleType without a Name attribute

I think the implementation relies heavily on the state. In my XSD I have a complexType, a sequence, an element(with no type attribute), a simpleType, a restriction and two enumerations in hierarchical order.
After the complexType "ends" there will be simpleTypes that define some of the elements i have inside the previous complexType.
The issue is that when I check the implementation, if the SimpleType Length and the Element length are larger than zero, I will .Pop() the SimpleType and then once I reach the SimpleType closing tag, I already did a .Pop() on the SimpleType so that simpleType that is outside will never be added to the opt.ProtoTree. Is this a wanted behaviour? It feels to me that the opt.Element should be empty after I processed the closing tag for complexType


overwrite panic during recovery

Panic message: attempt to overwrite compacted entries in xx.
This panic should be elided during parallel recovery. Consider this case:
file A: [index=1][index=2]
file B: [index=3][index=2][index=3]
It would raise a false panic if we recovery file B alone and try to merge it with file A afterwards.
Workaround:

Disable parallel recovery by setting recovery_threads to 1
Repair the log files with this script:

fn filter_append(id, first, count, rewrite_count, queue, ifirst, ilast) {
  if id == rid && first + count - 1 >= ifirst {
    return 2;  // discard existing
  }
  0
}
fn filter_compact(id, first, count, rewrite_count, queue, compact_to) {
  0
}
fn filter_clean(id, first, count, rewrite_count, queue) {
  0
}

/label affects-5.4
/label affects-6.0
/label affects-6.1


weird warning msg from PyCharm IDE

Before submitting a new issue, please check if a similar issue has already been filed.
Issue subject
So i found this warning from PyCharm, the code is copied from example folder.

Expected behaviour
Describe what should happen.
Should be no warnings.
Actual behaviour
Describe what happens instead. Please include the complete failure message.
Two warnings message happen, don't know why.
Steps to reproduce
Provide steps to let us know how we can reproduce this issue.
Pycharm new python file, and copy code from: examples/spot/market/depth.py
Environment
Provide any relevant information about your setup, such as:

Version of binance-connector-python  v3.3.1
Python version  3.11.4
Operating system Windows 11
Pycharm 2023.2.1

that's strange, are you able to run the example files?

that's strange, are you able to run the example files?

Yes, it runs  perfectly ... just these warnings, out of nowhere ...

Please open a new ticket if we can have more details related to the IDE.


mising function declaration for C99

Error when running: make
cache.c:25754:23: error: implicit declaration of function 'resize_configs_are_equal' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
} else if ( ! resize_configs_are_equal(&test_auto_size_ctl, \
Seem that function declaration prototype is missing.

Hi,
cache.c is not a nanopolish source file so I’m not sure where this error came from. Can you post the entire build log and the OS/compiler you’re using?
Jared

On Dec 19, 2021, at 9:33 AM, Yuliya @.***> wrote:

Error when running: make
cache.c:25754:23: error: implicit declaration of function 'resize_configs_are_equal' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
} else if ( ! resize_configs_are_equal(&test_auto_size_ctl, \
Seem that function declaration prototype is missing.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are subscribed to this thread.

Hi, thanks for your reply.
Using Mac with M1 chip running Big Sur.
Below is the output from make.
Thank you!!
make_output.txt

Thanks for the quick reply. It looks like the error is in libhdf5. I am off on holidays now but will look into this in January (hopefully).


Recommended pre-processing method?

Howdy, thanks for the repo. Was curious what type of rescaling or normalization you find works best with the Custom U-Net?

Hey @JordanMakesMaps , glad you like it! :)
As for normalization I honestly haven't experimented around it with this network. For my projects I just normalize pixel values between 0 and 1, same as in the example notebooks


Return None from lbp.poolData and xyk.poolAssets when pool doesn't exists

Actual status
Currently, if you call either of the aforementioned storage queries with a manually constructed poolId, it will return a set of default values such as:
xyk:
[
  0
  0
]

lbp:
{
  owner: bXgbR3K8BpV7Dn9imvyriviptpwAsaQyq1W6GLa77FJVqdyja
  start: 0
  end: 0
  assets: [
    0
    0
  ]
  initialWeight: 0
  finalWeight: 0
  weightCurve: Linear
  fee: {
    numerator: 2
    denominator: 1,000
  }
  feeCollector: bXgbR3K8BpV7Dn9imvyriviptpwAsaQyq1W6GLa77FJVqdyja
}

You can construct a non-existent poolId using the RPC for both xyk/lbp using the following RPC: getPoolAccount(assetA, assetB).
Expected status
I'd expect a None of an option<..> type to be returned in case the given poolId does not actually exist in the storage.

[ ] #251
[ ] Return None from xyk.poolAssets

@maht0rz #251 should be fixed in #231 already


Handle second SIGINT

Applications often listen to additional interrupt signals after the first one and terminate the application (with exit code 2) if it fails to gracefully stop after the first signal.
Do you think it would make sense in run as well?

Huh, I've never seen or heard about this. Do you have something I can read about it?

It would be easy to implement yourself, though...
{
    ctx, cancel := context.WithCancel(context.Background())
    g.Add(func() error {
        c := make(chan os.Signal, 1)
        signal.Notify(c, syscall.SIGUSR1)
        select {
        case <-c:
            return ErrUSR1
        case <-ctx.Done():
            return ctx.Err()
        }
    }, func(error) {
        cancel()
    })
}
{
    ctx, cancel := context.WithCancel(context.Background())
    g.Add(func() error {
        c := make(chan os.Signal, 1)
        signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
        select {
        case <-c:
            return ErrINT
        case <-ctx.Done():
            return ctx.Err()
        }
    }, func(error) {
        cancel()
    })
}
err := g.Run()
switch {
case errors.Is(err, ErrINT):
    os.Exit(1)
case errors.Is(err, ErrUSR1):
    os.Exit(2)
case err != nil:
    os.Exit(3)
case err == nil:
    os.Exit(0)
}

Do you have something I can read about it?

I've actually observed this behavior in a bunch of tools recently. For example, most of Hashicorp's tooling, like Terraform works like that. I guess it makes sense when there is a potentially a long shutdown period.
I think your example is slightly different from what I wanted to describe. The idea is that a second signal would immediately terminate the application (i.e. call os.Exit) terminating any long running shutdown procedure.
TBH, I've implemented it and it turned out to be less useful than I thought, so I don't use it anymore. (It seemed useful in a development environment, but graceful termination doesn't really matter in dev, so I can just kill it and move on)

a second signal would immediately terminate the application (i.e. call os.Exit) terminating any long running shutdown procedure

Ah, as in a second ctrl-C during a shutdown process would terminate the shutdown process immediately?

Yes

So basically the idea is
package main

import (
	"context"
	"log"
	"os"
	"os/signal"
	"time"

	"github.com/oklog/run"
)

func main() {
	log.Printf("starting...")
	ctx, cancel := context.WithCancel(context.Background())
	go func() {
		c := make(chan os.Signal, 1)
		signal.Notify(c, os.Interrupt)
		first := <-c
		log.Printf("got %s, sending cancel signal...", first)
		cancel()
		second := <-c
		log.Printf("got %s again, terminating immediately", second)
		os.Exit(2)
	}()
	log.Print(exec(ctx))
}

func exec(ctx context.Context) error {
	var g run.Group
	g.Add(run.SignalHandler(ctx))
	err := g.Run()
	log.Printf("error: %v; waiting 3s then returning...", err)
	time.Sleep(3 * time.Second)
	return nil
}

Can you suggest a way to provide that functionality cleanly in the module?

Can't say I see a "clean" way to do it. Best guess is a second groutine inside the signal handler func that continues listening to the signal channel.
But it should probably be optional, so it would require configuration....
I realized that I primarily wanted this feature for local development, because orchestrators will send a kill signal after a certain time anyway.

Here's a slightly more realistic example...
package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/oklog/run"
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	go func() {
		c := make(chan os.Signal, 1)
		signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
		first := <-c
		log.Printf("\ngot first signal (%s) - soft shutdown", first)
		cancel()
		second := <-c
		log.Printf("\ngot second signal (%s) - hard shutdown", second)
		os.Exit(1)
	}()
	log.Printf("starting...")
	err := exec(ctx)
	log.Printf("done with err=%v", err)
}

func exec(ctx context.Context) error {
	var g run.Group

	// subctx will get canceled when this actor's interrupt func is called, or
	// the ctx passed to exec is canceled. And we've set it up so that parent
	// ctx is canceled on ctrl-C.
	subctx, cancel := context.WithCancel(ctx)
	g.Add(func() error {
		<-subctx.Done()
		return subctx.Err()
	}, func(error) {
		cancel()
	})

	server := &http.Server{
		Addr:    "localhost:8080",
		Handler: http.HandlerFunc(handle),
	}

	g.Add(func() error {
		log.Printf("ListenAndServe...")
		err := server.ListenAndServe()
		log.Printf("ListenAndServe returned with err=%v", err)
		return err
	}, func(err error) {
		log.Printf("server received interrupt request with err=%v", err)
		grace := 3 * time.Second
		ctx, cancel := context.WithTimeout(context.Background(), grace)
		log.Printf("Shutdown initiated with grace period %s...", grace)
		err = server.Shutdown(ctx)
		log.Printf("Shutdown returned with err=%v", err)
		cancel()
	})

	return g.Run()
}

func handle(w http.ResponseWriter, r *http.Request) {
	d := r.URL.Query().Get("d")
	delay, _ := time.ParseDuration(d)
	if delay == 0 {
		delay = time.Second
	}

	log.Printf("%s: %s: %s: start", r.RemoteAddr, r.URL, delay)

	select {
	case <-time.After(delay):
		log.Printf("%s: %s: %s: elapsed", r.RemoteAddr, r.URL, delay)
	case <-r.Context().Done():
		log.Printf("%s: %s: %s: canceled", r.RemoteAddr, r.URL, delay)
	}
}

The critical thing being that the signal.Notify is done above the group and transformed into a context cancelation, and the cancelation is what the group actually sees. I could envision adding some kind of helper functionality to package run that provided this scaffolding, e.g.
package run

func Interruptable(ctx context.Context, onFirst, onSecond func(os.Signal), signals ...os.Signal) context.Context {
	ctx, cancel := context.WithCancel(ctx)
	go func() {
		c := make(chan os.Signal, 1)
		signal.Notify(c, signals...)
		onFirst(<-c)
		cancel()
		onSecond(<-c)
		os.Exit(2)
	}()
	return ctx
}

But that feels too... rigid? Why just two phases? Shouldn't the os.Exit be user-configurable instead of hard-coded? So then it probably should generalize to something like e.g.
package run

type PhaseFunc func(sig os.Signal, cancel func())

func Interruptable(ctx context.Context, signals []os.Signal, phases []PhaseFunc) context.Context {
	ctx, cancel := context.WithCancel(ctx)
	go func() {
		c := make(chan os.Signal, 1)
		signal.Notify(c, signals...)
		for _, phase := range phases {
			phase(<-c, cancel)
		}
	}()
	return ctx
}

Which would then need to be used like
func main() {
	var (
		signals = []os.Signal{syscall.SIGINT, syscall.SIGTERM}
		soft    = func(sig os.Signal, cancel func()) { log.Printf("got %s, soft shutdown...", sig); cancel() }
		hard    = func(sig os.Signal, _ func())     { log.Printf("got %s, hard shutdown...", sig); os.Exit(2) }
		phases = []run.PhaseFunc{soft, hard}
		ctx     = run.Interruptable(context.Background(), signals, phases)
	)
	log.Printf("starting...")
	err := exec(ctx)
	log.Print("done with err=%v", err)
}

But that doesn't feel like it adds much value versus the original code to me.

The way I normally achieve this behavior is by calling signal.Stop(c) after the first signal is received on c.

Won't that cause a second ctrl-C to just hard-kill the program?

That's right. I believe this is what this issue was originally about?
Although I probably misread that, because @sagikazarmark expected the second SIGINT to cause program to exit with code 2, and with hard-kill it will still exit with code 1.
I do agree with @sagikazarmark in that I too often see programs behave like that, and I think it's useful: first SIGINT begins graceful shutdown, and the second time you press ctrl+c it would force kill the program.


Computations requiring irregularly-chunked Zarr stores

All intermediate results in Cubed are written out to persistent storage via Zarr, but currently Zarr can't represent any chunked array, because the Zarr spec does not yet support irregular chunks.
This comes up in Cubed when a computation changes the chunking of an array from regularly chunked to irregularly chunked. An important example of such a computation is groupby, see https://github.com/tomwhite/cubed/issues/223.
I'm not sure if there are any other examples of array operations that might change regular chunks into irregular ones? np.pad comes to mind?

See discussion on ZEP003

You could implement pad in Cubed using concatenate, which already exists but has to copy the Zarr arrays to make them regularly-chunked. But if Zarr supported irregular chunking then it could be more efficient.


Strawberryfields with ProcessPoolExecutor

The number of shots can only be set to 1 for 'fock' backend, as described here. Therefore, to obtain 10000 measurement sampes, I need to run the circuit for 10000 times.  I have already defined the parameterized program and the engine as attributes of my class GBS . Then I tried to use ProcessPoolExecutor to to run the circuit 10000 times concurrently. However, this error occurs:  TypeError: LocalEngine.__new__() missing 1 required positional argument: 'backend.
A MWE is shown as follows:
from concurrent.futures import ProcessPoolExecutor

import strawberryfields as sf
from strawberryfields import ops

class GBS:
    def __init__(self):
        prog = sf.Program(2)
        a = prog.params('a')
        with prog.context as q:
            ops.Dgate(a ** 2) | q[0]  # free parameter
            ops.Sgate(1) | q[1]
            ops.MeasureFock() | q

        self.prog = prog
        self.eng = sf.Engine('fock', backend_options={'cutoff_dim': 5})

    def test(self, param):
        result = self.eng.run(self.prog, args={'a': param})
        return result

    def run(self):
        with ProcessPoolExecutor(1) as executor:
            future = executor.submit(self.test, 1)
            future.result()

def main():
    gbs = GBS()
    gbs.run()

if __name__=='__main__':
    main()

How can I solve this?

When I instantiated the Engine in the test method rather than in the init method, it worked. Why this happens?
from concurrent.futures import ProcessPoolExecutor

import strawberryfields as sf
from strawberryfields import ops

class GBS:
    def __init__(self):
        prog = sf.Program(2)
        a = prog.params('a')
        with prog.context as q:
            ops.Dgate(a ** 2) | q[0]  # free parameter
            ops.Sgate(1) | q[1]
            ops.MeasureFock() | q

        self.prog = prog

    def test(self, param):
        eng = sf.Engine('fock', backend_options={'cutoff_dim': 5})
        result = eng.run(self.prog, args={'a': param})
        return result

    def run(self):
        with ProcessPoolExecutor(1) as executor:
            future = executor.submit(self.test, 1)
            future.result()

def main():
    gbs = GBS()
    gbs.run()

if __name__=='__main__':
    main()

Hey @DS-Liu! Is there a reason why you can't call test a bunch of times like this?
def main():
    gbs = GBS()
    results = []

    for _ in range(10):
        results.append(gbs.test(1))

I'm not super familiar with ProcessPoolExecutor, so I'm not sure why your first example wasn't working 🤔
Let me know if this helps!

I want to run the code concurrently instead of a for loop which is quite slow. That's why I use ProcessPoolExecutor. I can't figure out why the Engine instantiated in the main process won't work in the child process. But when instantiated in the child process, it works.

I had better luck with getting multiprocessing to at least work (see docs here: https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing):
class GBS:
    def __init__(self):
        prog = sf.Program(2)
        a = prog.params('a')
        with prog.context as q:
            ops.Dgate(a ** 2) | q[0]  # free parameter
            ops.Sgate(1) | q[1]
            ops.MeasureFock() | q

        self.prog = prog
        self.eng = sf.Engine('fock', backend_options={'cutoff_dim': 5})

    def test(self, param):
        result = self.eng.run(self.prog, args={'a': param})
        return result

def main():

    gbs = GBS()
    results = []
    num_runs = 50_000

    parallel_time = time.process_time()

    with Pool() as pool:
        # call the function for each item in parallel
        for _ in range(num_runs):
            results.append(gbs.test(1))

    parallel_time = time.process_time() - parallel_time

    serial_time = time.process_time()

    for _ in range(num_runs):
        results.append(gbs.test(1))

    serial_time = time.process_time() - serial_time

    print(parallel_time, serial_time)

if __name__=='__main__':
    main()

25.561596 26.430917

Unfortunately the speedup isn't that great when I ran it on my machine. Not sure if I'm using it properly, but maybe this will help 😄

Actually you haven't use Pool object in your code. It can be used by calling pool.map() function. There's no speedup since you're running it serially.

Ah! Well, clearly you know more about parallelizing code than me 😅. In any case, if you use multiprocessing (properly) does it help?


Basic running of Lammps

This PR contains a working wrapper which focuses on running LAMMPS using FILE-IO (i.e. we are using input files to control LAMMPS).
Specifically in this PR, we are limiting at first the scope of how the user can use the LAMMPS wrapper.   Only some specific LAMMPS configuration will be supported at first.  Additionally, this PR adds only support for particles (i.e the lammps-related implementation of the ABCParticleContainer interface only has particles.  bonds will be added in a later PR).
This is work-in-progress.  The following is the goal of this PR:

[x] wrapper : basic sd-related methods for just ParticleContainer (e.g. get_particle_container, iter_particle_containers)
[ ] wrapper:  subset of basic cm/sp/bc-related configuration
[x] particle container (only particles)
[ ] documentation

Not included in this PR (but to be done in subsequent PRs) are the integration steps but they have been noted as issues and will be the next steps (e.g. #12, #11, #10, #8)
Not to be done till later is:

More advanced cm/sp/bc-related configuration
Support of bonds in ParticleContainer
Support of Mesh (i.e.for walls in lammps)

This PR replaces #5 (which was a PR reopening #1 as it was inadvertently merged).  #5 wasn't showing all the files for comment.

@nathanfranklin, I have updated my comment.

@nathanfranklin, some time using travis-solo might help debugging some issue with .travis.yml https://github.com/jstasiak/travis-solo.

you should not use sudo for the install command.

@itziakos Thanks for the tip!

@itziakos, @mehdisadeghi , or @ any-other-simphony-developer:     if you all have time to take a look at this, that would be very helpful. My plan is to address any easy comments/problems in this PR and open up an issue if anything that would take me longer to address.
I am going to start tackling the integration steps (e.g. #12, #11, #10, #8) in separate PRs. I probably will focus on the documentation stuff first (#12 and #8).

@nathanfranklin, I had a quick look nothing hit me as problematic. Unfortunately this week is a little busy for me and I will not have much time to help much with the reviews.

@itziakos , thanks.


Display the margin of lead in the state header

Is your feature request related to an issue? Please cite the issue.
Since there are recounts associated with margins of final vote counts, the lead margin is an important piece of information.
Clearly describe the solution you'd like
Just display the margin percentage in the state header. Bonus would be the list the margin needed to avoid an automatic recount or a recount challenge for that state.
See: https://github.com/alex/nyt-2020-election-scraper/pull/265

The latest vote margin/differential is always listed on the newest batch, so why would it need to be duplicated in the header?

I think they are referring to the percentage margin, which is usually the number stipulated for automatic recounts. It's easy enough to add to IterationSummary and display in the state header as a percentage, if that is desired. I have that part completed locally, and am looking into the "bonus" of having a hover-infotext with the state's laws regarding recounts (if any).

Hi!
I would love to work on this issue. Can you help me with getting started?
Thanks.

@noobyogi0010 I think this was already completed here https://github.com/alex/nyt-2020-election-scraper/pull/265

@ryannerocha Okay, thanks.

We've added the vote share percentage to the table header.


coding instructions made consistent in Add ID Attributes to Bootstrap…

In Add ID Attributes to Bootstrap Elements
changed
Give the well on the right the id to
Give the well on the right the id

npm tests pass
total:     710
passing:   710
duration:  2s
closes #7189

:+1:

:+1:


simplify unqualified col name aliases

Which issue does this PR close?

Closes #3722 .
Rationale for this change

Use simple names in schema - use the function name in this example and do not add the parameters.
What changes are included in this PR?

Use simple names in schema - use the function name in this example and do not add the parameters.
Are there any user-facing changes?

The unqualified column name will be changed

@andygrove please check this PR

Not sure I'm a fan. What happens with the alias if you use a function on more than one column?
E.g. select trim(a), trim(b) from table?

Here is what postgres does:
postgres=# select * from foo;
 val | time
-----+------
 A   |  1.0
 B   |  2.0
 C   |  3.0
(3 rows)

postgres=# select trim(val), trim(val) from foo;
 btrim | btrim
-------+-------
 A     | A
 B     | B
 C     | C
(3 rows)

postgres=#

Here is what datafusion-cli does from master:
DataFusion CLI v13.0.0
❯ create table foo as select column1 as val, column2 as time from (values ('A', 1.0), ('B', 2.0), ('C', 3.0)) as sq;

0 rows in set. Query took 0.023 seconds.
❯ select trim(val), trim(val) from foo;
Plan("Projections require unique expression names but the expression \"trim(foo.val)\" at position 0 and \"trim(foo.val)\" at position 1 have the same name. Consider aliasing (\"AS\") one of them.")

What happens when using the same function more than once?

@Dandandan thanks for you comment, the engine historically fails if the alias are the same,  like if you run
let sql = "SELECT COALESCE(NULL, 'test'), COALESCE(NULL, 'test')";
it will fail even now.
supposingly we need a separate ticket to try to do postgres behaviuor, or add some number like
coalesce_0, coalesce_1. I'm happy to do this if needed

Ok, but the current branch now also fails with
select trim(a), trim(b) from table? As the two are now giving the same alias. This is my main worry to merge this. As suffix that @comphead suggests seems a solution.

Ok, but the current branch now also fails with select trim(a), trim(b) from table? As the two are now giving the same alias. This is my main worry to merge this. As suffix that @comphead suggests seems a solution.

This is correct, as we shortened the column name the collision will happen more often. But this gives us more motivation to make unique readable aliases.
Because currently user may not be happy to refer to column from outer sql to nested subquery which contains not friendly alias like btrim(test.a,Utf8(\"ab\"))

@alamb @Dandandan should I put unique col aliasing like trim_0, trim_1 as part of this PR, or create a new one?

@alamb @Dandandan should I put unique col aliasing like trim_0, trim_1 as part of this PR, or create a new one?

I am not sure -- perhaps @andygrove  can weigh in here as he filed https://github.com/apache/arrow-datafusion/issues/3722

@alamb @Dandandan should I put unique col aliasing like trim_0, trim_1 as part of this PR, or create a new one?

I am not sure -- perhaps @andygrove can weigh in here as he filed #3722

@andygrove please help here until the code rots

I will catch up on this and other pending PRs tomorrow.

Marking this as a draft as it is waiting on feedback and not ready for review or merge

Postgres allows duplicate column names in a schema but then throws an error if you try and select one of those columns. We currently fail on creating a schema with duplicate columns.
I guess the first question is, do we want to be more like Postgres here?

Postgres allows duplicate column names in a schema but then throws an error if you try and select one of those columns. We currently fail on creating a schema with duplicate columns.
I guess the first question is, do we want to be more like Postgres here?

Thanks @andygrove for the feedback.
Most likely we should avoid current naming for not qualified columns as its not user friendly and hard to use in outer subqueries.
And perhaps its a bigger topic to discuss naming standard, to be like a Postgres or something else, but we can create a discussion ticket on this first, to get all options from contributors.
If you ask me, I like how its done in Trino
select t.*, 'col_again' from ( 	select 'col', 2 a, count(1) ) t
_col0|a|_col2|_col3    |
-----+-+-----+---------+
col  |2|    1|col_again|

Closing as this PR is over a year old. Please feel free to reopen it / rebase it if you plan to keep working on it


Newlines are removed from multi-line strings

When formatting Groovy files, line breaks are removed from strings.
$ cat test.groovy
def s = '''

'''
$ npm-groovy-lint --format --verbose test.groovy
  2     fixed    File test.groovy has consecutive blank lines  ConsecutiveBlankLines

npm-groovy-lint results in  linted files:
┌─────────┬───────────┬─────────────┬─────────────┬─────────────────┐
│ (index) │ Severity  │ Total found │ Total fixed │ Total remaining │
├─────────┼───────────┼─────────────┼─────────────┼─────────────────┤
│    0    │  'Error'  │      0      │      0      │        0        │
│    1    │ 'Warning' │      0      │      0      │        0        │
│    2    │  'Info'   │      1      │      1      │        0        │
└─────────┴───────────┴─────────────┴─────────────┴─────────────────┘
Failure: 0 error(s) have been found
 0 warning(s) have been found
 1 info(s) have been found
$ cat test.groovy
def s = '''

'''

That's probably an upstream issue: https://github.com/CodeNarc/CodeNarc/issues/720

As you notified in https://github.com/CodeNarc/CodeNarc/issues/720, the error comes from the false positive trigger by CodeNarc
Once it will be solved in CodeNarc and when they'll release a new version, it will be upgraded in npm-groovy-lint and the problem will disappear :)

Not sure if an upstream fix will make this issue go away. IMHO, this issue is about the formatter breaking the code, or at least changing its behavior, which a mere formatter shouldn't.
Maybe the formatter should not be enabled by default.


In-memory representation of worms

Currently in the Python implementation, worm data is represented as a pandas DataFrame with the rows for time frames and the columns being a MultiIndex, with columns for each skeleton point in x and y for each worm id.
This tabular representation is memory-inefficient if the worm data is "sparse", e.g. if there are multiple worms over mutually exclusive time frames.
Another representation could be a dictionary:
worm1.data = {1:df1, 2:df2, 3,df3}
Where 1,2,3 are the worm ids and df1, df2, df3 are dataframes specific to those individual worms.
I've never dealt with multiworm data before so I'm getting issues with @Ichoran's worm data file which has 90 worms.  This would keep memory use down below 300 MB, whereas the current implementation uses 600-1200 MB, and more during the construction phase, which actually breaks Travis CI's 3 GB limit.
Does anyone have any thoughts or preferences here?

I strongly recommend some sort of dictionary.  You could imagine recordings that go on for days, and worms that are followed only for a few minutes.  Having vast tables of zeros isn't helping anything there.  You do want to have an index, though, so you can look up all IDs present at one timepoint.

OK I have done this now(as an ordered dictionary), with f20ac8a


[Bugs] UPF Not Responding to PFCP Session Deletion/Establishment Requests During High Data Throughput

Describe the bug

While sending/receiving data at UL: 100Mbps / DL: 100Mbps, if the UE is detached, the UPF does not send a response to the PFCP Session Deletion Request from the SMF.
Subsequently, responses to PFCP Session Establishment Requests are not sent, and the PDUSession cannot be established.
When the issue occurs and the UPF is stopped (pkill upf), the following log remains output and the UPF cannot be stopped:

[INFO][UPF][PFCP][LAddr:<IP_ADDRESS>:8805] Stopping pfcp server
To Reproduce
Steps to reproduce the behavior:

Establish a PDU Session.
Send/receive packets with UDP UL: 100Mbps / DL: 100Mbps.
Execute detachment while sending/receiving packets.

Expected behavior
The UPF should send a response to the PFCP Session Deletion (Establishment) Request, but from the logs during the UPF stop, it seems that there is an abnormality such as a stack issue in the PFCP receiver.
Environment (please complete the following information):

free5GC Version: v3.4.2
OS: Ubuntu 20.04.6 LTS
Kernel version: 5.4.0-125-generic
go version: Ubuntu go1.21.8 linux/amd64

Hi,
You can adjuste the following settings in smfcfg.yaml to reduce usage report from UPF.
  urrPeriod: 30 # default usage report period in seconds
  urrThreshold: 500000 # default usage report threshold in bytes

or even higher.

We have confirmed that the PFCP Session Deletion response was indeed missing. We will fix this issue as soon as possible.

If there are too many usage reports under high throughput conditions, the PFCP session deletion response may be lost. It is recommended that you adjust the usage report period as suggested above, and then you will be able to see the deletion response.

Thank you for your response.
However, no matter how much we increase the urrPeriod or urrThreshold, won't the same problem occur if even faster communication takes place for a longer duration?
Additionally, in that case, I think the impact of the UPF being unable to kill the process is significant.
Isn't it necessary to change the processing priority so that PFCP processing is handled with higher priority than UPF reporting?
I believe this issue should not be closed yet.


The bridge helmchart can only work with hostpath volumes

It should also be able to work with csi volumes

Modified it in d52b63fc8649cc4a2d5004fb8392859f3ae8be0d but when deploying there are errors about the diskquota being exceeded:
tftbscbridge-8fb4f5bff-5nlkm.log


MAVIS submodule

Missing #include <cstdint> at DecoderTypes
Which is fixed in a recent commit

@orion160 Can I take up this issue?

Hmmm feel free to submit a PR, though I am not a maintainer

Thanks, I will!
On Tue, 13 Aug 2024 at 8:34 PM, Isaac David @.***>
wrote:

Hmmm feel free to submit a PR, though I am not a maintainer
—
Reply to this email directly, view it on GitHub
https://github.com/riscv-software-src/riscv-perf-model/issues/196#issuecomment-2286481208,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A5WMIMRQIE3U7XRPQAY6RHLZRIOAPAVCNFSM6AAAAABMKMQAYOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBWGQ4DCMRQHA
.
You are receiving this because you commented.Message ID:
@.***>

I have submitted a PR [https://github.com/sparcians/mavis/pull/18] ..Pls check and suggest me about the feedback


[WIP] Implement agent management

Cluster mode view is not implemented yet.

Can one of the admins verify this patch?


Resolve security issue CVE-2022-38900

decode-uri-component 0.2.0 is vulnerable to Improper Input Validation resulting in DoS.
Reported by github Dependabot last year; high severity
bumped decode-uri-component to v0.2.2 which resolves th is issue

^0.2.0 already allows version 0.2.2. Why is this needed?

It may "allow" it, but the package.jon says 0.2.0 and that's what's included. Since other folks use this module, specifically upgrading to the version that doesn't have the security violation resolves the problem.

Do you mean that you have some tool that gives you warnings because source-map-resolve has the potential to install a vulnerable version? Shouldn’t it give you warnings based on what you actually install?
But – anyway. I find it boring to use my free time to do things with this deprecated package that I don’t like. It might be easy to fix this thing, but in a couple of months there will be some other vulnerability in some other dependency and the cycle repeats. Or someone finds a vulnerability in source-map-resolve itself. Not fun.
There is absolutely no reason to use source-map-resolve. It is deprecated. The things that depend on source-map-resolve are either deprecated, unmaintained or vulnerable, and they all have better alternatives.

So why don't you turn over maintenance to someone who can spend a tiny amount of time to accept fixes.
You created an open source module -- currently has 16,693,879 weekly downloads -- and you're unwilling to accept a PR to fix a security vulnerability?...Sad

Your PR does not fix a security vulnerability.
I suggest you fork the packages you use and maintain them.


Unit test cases might be added

There is a package to help build up test cases for grammars, which might be helpful for this project,
https://github.com/PanAeon/vscode-tmgrammar-test
With a good amount of test cases, regression can be easier.

Happy to put together a PR porting some of the test cases I’d put together for the Esbonio extension, if that would be useful 😄


Update Japanese message catalogue

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)

Proposed changes

This PR add Japanese translation of message catalogue according to recent changes to English messages.
Checklist

[x] I have read the contribution guidelines
[ ] For non-bugfix PRs, I have discussed this change on the forum/slack team.
[x] I have run grunt to verify the unit tests pass
[ ] I have added suitable unit tests to cover the new/changed functionality

Coverage decreased (-0.007%) to 67.566% when pulling c1d85f760de599f9c86ab1f07f8aa3abf929a4c9 on node-red-hitachi:update-message-jp into 88ad2f4c18487dc632e135f6dbc6e7c49526791d on node-red:dev.


XOrg crashing randomly

Issue description
What the title says: this typically happens about once a day. Appears to be distinct from #21026 based on the stack trace:
#0  0x00007faa7867d0f3 in __GI__IO_default_xsputn () from /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/libc.so.6
#1  0x00007faa78670c6d in __GI__IO_padn () from /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/libc.so.6
#2  0x00007faa786504d0 in vfprintf () from /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/libc.so.6
#3  0x00007faa786fe7c6 in __vsnprintf_chk () from /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/libc.so.6
#4  0x00007faa786fe728 in __snprintf_chk () from /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/libc.so.6
#5  0x00007faa781dcb27 in pci_device_vgaarb_set_target () from /nix/store/ix3l6ys32kxjlljg64nkyc03az337sp9-libpciaccess-0.13.4/lib/libpciaccess.so.0
#6  0x000000000049273c in VGAarbiterSpriteMoveCursor ()
#7  0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#8  0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#9  0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#10 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#11 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#12 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#13 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#14 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#15 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#16 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#17 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#18 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#19 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#20 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#21 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#22 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#23 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#24 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()
#25 0x0000000000492753 in VGAarbiterSpriteMoveCursor ()

(and so on for many more frames, so I'm guessing it's a stack corruption bug?). I have a core dump but I'm not posting it because I'm not sure if it might contain sensitive information — if anyone can advise me concerning that it would be much appreciated!
Steps to reproduce
Unknown.
Technical details

System: 17.03.1052.aa0383397a (Gorilla)
Nix version: nix-env (Nix) 1.11.8
Nixpkgs version: "17.03.1052.aa0383397a"

We can try this update: https://lists.x.org/archives/xorg-announce/2017-April/002791.html – perhaps this point will help:

vgaarb: add a the trailing NULL character on read(vgaarb_fd)

I haven't encountered this any more, and nobody else has confirmed this, so I'll close it.


Bring back the -c -iv parameters in command line.

The command line and the parameters is very essential for linux users.
https://github.com/PokemonGoF/PokemonGo-Bot/commit/647bac8322dcebcccc116c81343d8378445236ff
I'd rather prefer using the command line. A lot of people are confused of the "new way". Please add the old way back and make it default, also add an option inside config.json to enable the release_config.json for those who rather want to use that.

As stated in Slack:
It seems like as this bot grows, it is going to be more and more configurable. At a certain point, it doesn't make sense to configure everything through cli args and will have to move to something like json configuration.
If the problem is that for multiple bots, the only real way is via command line args, we can probably find a way to fix that.
I could imagine a cli flag that passed a path to a config file. Would that solve the problem at hand?
Perhaps an approach would be to pass either a cli arg that tells it what config file to use, or there is a command line arg that take a string of the json it should use as config

@TheSavior Sure, but if people wants to use the parameters, make the parameters (i.e. -c -iv) overwrite the config file.

I think having both would just add more functionality, if someone wants to run 3-4 accounts at once using .bat files and parameters it is easier than having 4 separate bots

I think that is reasonable, but how would you handle overwriting deeply nested configuration? For example, if we merged release_config into config and you wanted to overwrite a field like this: https://github.com/TheSavior/PokemonGo-Bot/blob/0d44885fcd8d710e54b718d3468094965801d56d/config.json.example#L18
Perhaps
--release_config.any.release_under_cp 300

@TheSavior Great statement. How about --release_config.any.cp 200 and --release_config.any.iv 300?

One of the downsides of that approach is that it may not make sense to support the shorthand cli args anymore so that the configuration can be defined in only one place config.json and it becomes clear in code how they are being overridden. What are your thoughts on that?
The result would be that it would be very clear what the command line flags are, because they match the config.json perfectly, but it would mean that the length of the command to run would be longer.

@TheSavior What do you recommend then?

Here is what I propose:
We check in a config.json file by default. It contains the current defaults that are set in pokecli.py
We remove all of the custom command line arguments that currently exist.
We create a new command line flag. --config_file. If this is set, we will read that file for default configuration. If it is not set, we read config.json. If there is no config.json, we throw.
We then dynamically read the rest of the command line flags. We take the format like --release_config.any.cp 200 and use that to overwrite this key in the loaded configuration:
{
  "release_config": {
    "any": {
      "cp": 300
    }
  }
}

If there is no key, or no path to that key, for example --release_config.any.doesnt_exist 'foo' or --release_config.doesnt_exist, then we throw.
Down the road, if this gets unmanageable and people want something different, we add an additional command line argument called --config. This argument takes JSON and is merged on top of the loaded configuration file.

@TheSavior I like this way. I would rather do it in this approach.

@solderzzc and @douglascamata Can you chime in with your concerns or support for this proposal?
Also, can you rename the ticket to something like "Proposal: Enable multiple configuration files and overriding from CLI"

Sure, let me do it.

Config files are fine, but result in duplicate data when dealing with multiple accounts.
For reusability's sake, re-implement at least -u, -p and -a.
Other cli parameters are a luxury, but they can still be useful if you don't feel like creating an entire separate JSON file for one small config change.

This feels like a discussion that shouldn't be happening.
Configuration files allow for a lot of configuration without much hassle or a million command line parameters.
Command line allows for easy access, overwriting and simpler automation.
Both have their valid uses. Load config.json by default, let users overwrite whatever they want via command line.

@sebastienvercammen is completely correct.
Im more than happy to try reimplement both of these - as someone who is multi-boxing this installation it becomes a pain running several folders.

@sebastienvercammen, sounds like you agree with the proposal then. 👍

This issue has been resolved, closing this for now.
/closed