change how strings are handled n OptiX 7

Description
This patch changes how strings are handled in the OptiX 7 codepath.  Strings are now handled as hashes which are computed at compile time.  This means a few things:

strings are no longer copied to the GPU
we no longer have to create a special CUDA file that contains string pointers (which removes the libnvrtc dependency -- hooray!)
strings are now hashed with a constexpr hash function which requires c++14.  Because we don't have CUDA programs that depend on runtime values (ie. pointers to GPU memory),  I'm hoping compilation should be smoother overall.
osl_printf() output is now handled by the CPU.  osl_printf() copies shaders' printf() output to a GPU buffer.  After rendering, the CPU reads this buffer and handles all the output.  This means that all strings in the GPU code need to be "registered" with the host program.   See testshade and testrender for the appropriate changes.
because we no longer have a run-time compiled CUDA module to store global variables, I've now added extra optixLaunch() calls to testshade and testrender to set global variables via the pipeline launch parameters.

The OptiX 6 codepath should be untouched.
Tests
Because strings are no longer copied to the CPU, we don't have access to their lengths nor can we reference individual characters.  As such, the testoptix.optix test now fails due to differing text output (the images are fine).
Checklist:

[ ] I have read the contribution guidelines.
[ ] I have previously submitted a Contributor License Agreement.
[ ] I have updated the documentation, if applicable.
[ ] I have ensured that the change is tested somewhere in the testsuite (adding new test cases if necessary).
[ ] My code follows the prevailing code style of this project.

Other than my questions this LGTM!

This LGTM. Do you want to wait for @fpsunflower (whoever that is) to look at it? Or move the constexpr hash to oiio before checkin?

I'll look today as well. Maybe @timgrant has a chance to glance at it also?

Very cool. I will look this over ASAP, hopefully tomorrow.

Overall, this change looks good to me. Maintaining printf() is clutch. Very nice.
I do worry a little bit about the potential loss of functionality, but I'm confident that there should be some way to make the string contents available on the device, if that should prove to be necessary. We can cross that bridge if we get to it.
One last comment, I notice that the string table is still being constructed for O7, but as far as I can tell it's not used for anything.

Addressing more comments, removed string table from OptiX7 code-path, and removed unnecessary 'hit' programs from the "setglobals" launch.

Any other feedback on this?

Any other feedback on this?

I wanted to walk through out a "possible" alternative to the hash.  I might be missing some of the requirements, but let me outline an approach and see what I'm missing.
Conditions:

At rendertime enough shader network parameters have been set or connected such that any string operations in OSL shader networks can be constant folded and all ustring values are known by the time shader optimization finishes.  T

Requirements:

Device side compile time constant value for a DeviceString be available for all STRING_PARAM() entries.
DeviceString be equal and not equal comparable on the device.
At rendertime JIT, additional ustrings can be mapped to DeviceString values that can be embedded in JIT as LLVM IR constants.
On host, DeviceString value can be used to lookup a ustring (in order to process printf's, texture filenames, etc.)

Proposal:
Let DeviceString encapsulate a 64bit unsigned integer.
Use strdecls.h with some macro magic to populate constexpr DeviceString for DeviceStrings namespace with a linearly increasing index.  IE:
namespace DeviceStrings {
    __device__ OSL_NAMESPACE::DeviceString _emptystring_(0u);
    __device__ OSL_NAMESPACE::DeviceString camera(1u);
    __device__ OSL_NAMESPACE::DeviceString common(2u);
   // ...
    __device__ OSL_NAMESPACE::DeviceString unknown(127u);
}

The same initialization would happen on the host side.
In addition on the host side where rendertime jit code gen happens we need a way to dynamically create additional device strings dynamically from a ustring:
namespace { // Unnamed
struct DeviceStringManager {
    std::vector<ustring> m_ustring_by_device_string_index;
    std::unordered_map<ustring, uint64_t, ustringHash>m_device_string_index_by_ustring;
public:
    DeviceStringTable():m_device_count(0) {}

    DeviceString register_device_string(ustring val) {
        auto iter = m_device_string_index_by_ustring.find(val);
        if (iter == m_device_string_index_by_ustring.end()) {
            uint64_t device_string_index = m_ustring_by_device_string_index.size();
            m_ustring_by_device_string_index.push_back(val);
            iter = m_device_string_index_by_ustring.insert(std::make_pair(val, device_string_index)).first;
        }
        return DeviceString((*iter).second);
    }

    ustring lookup_ustring(DeviceString val) {
        OSL_ASSERT(static_cast<uint64_t>(val) < m_ustring_by_device_string_index.size());
        return m_ustring_by_device_string_index[static_cast<uint64_t>(val)];
    }
}

DeviceStringManager &get_device_string_manager() {
    static DeviceStringManager  lazy_manager;
    return lazy_manager;
}
} // namespace

DeviceString register_device_string(ustring val) {
    return get_device_string_manager().register_device_string(val);
}

ustring lookup_ustring(DeviceString val) {
    return get_device_string_manager().lookup_ustring(val);
}

Additionally on the host side, process strdecls.h again.  This time to register all the StringParams and validate they match
OSL_ASSERT(register_device_string("") == DeviceStrings::_emptystring_);
OSL_ASSERT(register_device_string("camera") == DeviceStrings::camera);
OSL_ASSERT(register_device_string("common") == DeviceStrings::common);
// ...
OSL_ASSERT(register_device_string("unknown") == DeviceStrings::unknown);

This could be done in a static object constructor so it happens only once.
So because strdecls is processed in the same order at compile time on the device and execution time registering device strings on the host, we have a set of comparable StringParams between the host and device.  Furthermore any new dynamically registered DeviceStrings will map to the existing device strings or create new unique entries with 0 collision potential.
And we still have the efficiency on the device side of using compile time constants.
Might suggest changing it from DeviceString to StringId (or something implying its just an identifier or handle, not a string itself).

I wanted to walk through out a "possible" alternative to the hash.  I might be missing some of the requirements, but let me outline an approach and see what I'm missing.
Conditions:

At rendertime enough shader network parameters have been set or connected such that any string operations in OSL shader networks can be constant folded and all ustring values are known by the time shader optimization finishes.  T

Requirements:

Device side compile time constant value for a DeviceString be available for all STRING_PARAM() entries.
DeviceString be equal and not equal comparable on the device.
At rendertime JIT, additional ustrings can be mapped to DeviceString values that can be embedded in JIT as LLVM IR constants.
On host, DeviceString value can be used to lookup a ustring (in order to process printf's, texture filenames, etc.)

Proposal:
Let DeviceString encapsulate a 64bit unsigned integer.
Use strdecls.h with some macro magic to populate constexpr DeviceString for DeviceStrings namespace with a linearly increasing index.  IE:
namespace DeviceStrings {
    __device__ OSL_NAMESPACE::DeviceString _emptystring_(0u);
    __device__ OSL_NAMESPACE::DeviceString camera(1u);
    __device__ OSL_NAMESPACE::DeviceString common(2u);
   // ...
    __device__ OSL_NAMESPACE::DeviceString unknown(127u);
}

The same initialization would happen on the host side.
In addition on the host side where rendertime jit code gen happens we need a way to dynamically create additional device strings dynamically from a ustring:
namespace { // Unnamed
struct DeviceStringManager {
    std::vector<ustring> m_ustring_by_device_string_index;
    std::unordered_map<ustring, uint64_t, ustringHash>m_device_string_index_by_ustring;
public:
    DeviceStringTable():m_device_count(0) {}

    DeviceString register_device_string(ustring val) {
        auto iter = m_device_string_index_by_ustring.find(val);
        if (iter == m_device_string_index_by_ustring.end()) {
            uint64_t device_string_index = m_ustring_by_device_string_index.size();
            m_ustring_by_device_string_index.push_back(val);
            iter = m_device_string_index_by_ustring.insert(std::make_pair(val, device_string_index)).first;
        }
        return DeviceString((*iter).second);
    }

    ustring lookup_ustring(DeviceString val) {
        OSL_ASSERT(static_cast<uint64_t>(val) < m_ustring_by_device_string_index.size());
        return m_ustring_by_device_string_index[static_cast<uint64_t>(val)];
    }
}

DeviceStringManager &get_device_string_manager() {
    static DeviceStringManager  lazy_manager;
    return lazy_manager;
}
} // namespace

DeviceString register_device_string(ustring val) {
    return get_device_string_manager().register_device_string(val);
}

ustring lookup_ustring(DeviceString val) {
    return get_device_string_manager().lookup_ustring(val);
}

Additionally on the host side, process strdecls.h again.  This time to register all the StringParams and validate they match
OSL_ASSERT(register_device_string("") == DeviceStrings::_emptystring_);
OSL_ASSERT(register_device_string("camera") == DeviceStrings::camera);
OSL_ASSERT(register_device_string("common") == DeviceStrings::common);
// ...
OSL_ASSERT(register_device_string("unknown") == DeviceStrings::unknown);

This could be done in a static object constructor so it happens only once.
So because strdecls is processed in the same order at compile time on the device and execution time registering device strings on the host, we have a set of comparable StringParams between the host and device.  Furthermore any new dynamically registered DeviceStrings will map to the existing device strings or create new unique entries with 0 collision potential.
And we still have the efficiency on the device side of using compile time constants.
Might suggest changing it from DeviceString to StringId (or something implying its just an identifier or handle, not a string itself).

I think I might have missed a requirement here.
Is it required that for a JIT'd shader that the same DeviceString(StringID) value be produced the next time the shader is JIT'd?
I suppose if an underlying driver wanted to cache binary version of a shader and match it up next run (assuming the same LLVM IR would be produced and could be matched to a previously built binary version) there could be a problem.  The StringID approach doesn't guarantee any dynamicall created ID's will be in the same order, only the StringParams would be consistent (same ID's) between runs, whereas a hashing approach would make such a guarantee.   Is this a realworld requirement?

I think I might have missed a requirement here.
Is it required that for a JIT'd shader that the same DeviceString(StringID) value be produced the next time the shader is JIT'd?
I suppose if an underlying driver wanted to cache binary version of a shader and match it up next run (assuming the same LLVM IR would be produced and could be matched to a previously built binary version) there could be a problem.  The StringID approach doesn't guarantee any dynamicall created ID's will be in the same order, only the StringParams would be consistent (same ID's) between runs, whereas a hashing approach would make such a guarantee.   Is this a realworld requirement?

@AlexMWells Yes, indeed. The post-JIT, NVIDIA driver-side compilation of the PTX and for that matter the Cuda of the renderer-side components to the actual device code and other things OptiX does with that block of code are incredibly,  painfully expensive. This is ameliorated by Optix's compile cache. So subsequent runs of the renderer can go much faster as long as the generated PTX/Cuda is absolutely identical. (Eric Enderton will, I'm sure correct any botched terminology or description I have given. I'm doing a little hand waving without consulting my notes.)
So the reason why we couldn't use the CPU-side address as the representation of the string on the Cuda side is that these addresses, while guaranteed to be unique, are not going to be identical from run to run, so we don't want them showing up anywhere in the generated PTX or it will foil the cache. (For now, it doesn't matter that the addresses would be host memory, because on the GPU side we are taking for granted that by the time we get on the other side of runtime optimization, we are only doing string assignment, ==, and != comparisons, all of which require only knowing the pointer value, not having access to the characters it points to.)
Using a unique integer index for each string doesn't really get us any farther, because the correspondence of these IDs to the strings is dependent on the order that they are created, which can vary from scene to scene and run to run.
Another choice, and what we started with but thought was too big a PITA, is essentially to have a variable on the Cuda side for each ustring that will be used. Fill in the value (holding the host-side address) at runtime, but the PTX/Cuda thinks it's using a variable, so it's not polluted with the run-to-run-nonconstant value. But this requires a lot of coordination between the Cuda side, OSL internals, and the renderer itself to make sure all strings are predeclared so that those variables are set up. The burden of coordinating with the renderer was something we really wanted to avoid.
Hence, the hope that the hashes would be good enough. Because the hashes are stable from run to run for a particular string, it requires no pre-declaration, no allocation of extra variables, they are just constants to the PTX, but constants that are always the same for a given string, so the compile cache is happy. The only problem is what happens if there is a hash collision (and specifically, a collision between two things that will actually be compared -- a collision between a color space name, and a texture wrap mode name, say, is of no consequence).

@AlexMWells Yes, indeed. The post-JIT, NVIDIA driver-side compilation of the PTX and for that matter the Cuda of the renderer-side components to the actual device code and other things OptiX does with that block of code are incredibly,  painfully expensive. This is ameliorated by Optix's compile cache. So subsequent runs of the renderer can go much faster as long as the generated PTX/Cuda is absolutely identical. (Eric Enderton will, I'm sure correct any botched terminology or description I have given. I'm doing a little hand waving without consulting my notes.)
So the reason why we couldn't use the CPU-side address as the representation of the string on the Cuda side is that these addresses, while guaranteed to be unique, are not going to be identical from run to run, so we don't want them showing up anywhere in the generated PTX or it will foil the cache. (For now, it doesn't matter that the addresses would be host memory, because on the GPU side we are taking for granted that by the time we get on the other side of runtime optimization, we are only doing string assignment, ==, and != comparisons, all of which require only knowing the pointer value, not having access to the characters it points to.)
Using a unique integer index for each string doesn't really get us any farther, because the correspondence of these IDs to the strings is dependent on the order that they are created, which can vary from scene to scene and run to run.
Another choice, and what we started with but thought was too big a PITA, is essentially to have a variable on the Cuda side for each ustring that will be used. Fill in the value (holding the host-side address) at runtime, but the PTX/Cuda thinks it's using a variable, so it's not polluted with the run-to-run-nonconstant value. But this requires a lot of coordination between the Cuda side, OSL internals, and the renderer itself to make sure all strings are predeclared so that those variables are set up. The burden of coordinating with the renderer was something we really wanted to avoid.
Hence, the hope that the hashes would be good enough. Because the hashes are stable from run to run for a particular string, it requires no pre-declaration, no allocation of extra variables, they are just constants to the PTX, but constants that are always the same for a given string, so the compile cache is happy. The only problem is what happens if there is a hash collision (and specifically, a collision between two things that will actually be compared -- a collision between a color space name, and a texture wrap mode name, say, is of no consequence).

I wonder how do you ensure that strings from different compilation units
get different ids. Or the same one if they are equal.
On Mon, 11 Jan 2021 at 20:41, Alex Wells<EMAIL_ADDRESS>wrote:

I think I might have missed a requirement here.
Is it required that for a JIT'd shader that the same
DeviceString(StringID) value be produced the next time the shader is JIT'd?
I suppose if an underlying driver wanted to cache binary version of a
shader and match it up next run (assuming the same LLVM IR would be
produced and could be matched to a previously built binary version) there
could be a problem. The StringID approach doesn't guarantee any dynamicall
created ID's will be in the same order, only the StringParams would be
consistent (same ID's) between runs, whereas a hashing approach would make
such a guarantee. Is this a realworld requirement?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-758178727,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24CRSAXCDIF6CH74YSTSZNH7DANCNFSM4UOAKVPQ
.

I wonder how do you ensure that strings from different compilation units
get different ids. Or the same one if they are equal.
On Mon, 11 Jan 2021 at 20:41, Alex Wells<EMAIL_ADDRESS>wrote:

I think I might have missed a requirement here.
Is it required that for a JIT'd shader that the same
DeviceString(StringID) value be produced the next time the shader is JIT'd?
I suppose if an underlying driver wanted to cache binary version of a
shader and match it up next run (assuming the same LLVM IR would be
produced and could be matched to a previously built binary version) there
could be a problem. The StringID approach doesn't guarantee any dynamicall
created ID's will be in the same order, only the StringParams would be
consistent (same ID's) between runs, whereas a hashing approach would make
such a guarantee. Is this a realworld requirement?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-758178727,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24CRSAXCDIF6CH74YSTSZNH7DANCNFSM4UOAKVPQ
.

@aconty , for the StringID approach the StringParams:: are the only ones that would have same ID's because the result of the macro magic just produces DeviceStrings with hard coded #'s and only on the host side are those actually registered as strings (and validated to have the correct #s.  Thus precompiled code (the user's of StringParams) would match between multiple compilation units on host or device.
Alas, it would be of no help for the JIT'd dynamic strings, unless the DeviceStringManager became persistent.

@aconty , for the StringID approach the StringParams:: are the only ones that would have same ID's because the result of the macro magic just produces DeviceStrings with hard coded #'s and only on the host side are those actually registered as strings (and validated to have the correct #s.  Thus precompiled code (the user's of StringParams) would match between multiple compilation units on host or device.
Alas, it would be of no help for the JIT'd dynamic strings, unless the DeviceStringManager became persistent.

I think we have discussed the same design alternatives often enough that it would be worth writing a document summarizing the pros and cons of each and why we chose the current hashing strategy.
It doesn't have to be the final word on the topic, but it would help anyone trying to wrap their heads around why this approach was chosen over other ones.

I think we have discussed the same design alternatives often enough that it would be worth writing a document summarizing the pros and cons of each and why we chose the current hashing strategy.
It doesn't have to be the final word on the topic, but it would help anyone trying to wrap their heads around why this approach was chosen over other ones.

@etheory reports that he is able to generate hash collisions. It's scary that they are "short", but also they are kinda random looking.
I think it's an open question whether we need to fix this right away, or whether we can let the hash approach stand for now, knowing that we'll need to come back and fix this for real, but in the mean time we can more carefully ponder how to do it without running afoul of the ptx cache problem.
Opinions?

@etheory reports that he is able to generate hash collisions. It's scary that they are "short", but also they are kinda random looking.
I think it's an open question whether we need to fix this right away, or whether we can let the hash approach stand for now, knowing that we'll need to come back and fix this for real, but in the mean time we can more carefully ponder how to do it without running afoul of the ptx cache problem.
Opinions?

As further background, we should spell out that there are three sources of strings, really:
(1) "pre-known" strings that we know that are referenced inside the OSL runtime implementation, for example all the ustrings in the Strings:: namespace (examples: "RGB", "snoise", "wrap", etc.).
(2) strings that are specified in the shader as literals, or come from the scene as parameter instance values (such as most texture names), or are constructed in the shader in a way that can resolved during runtime optimization of the shader group. In other words, all the strings not in group 1 that the shader itself will try to reference.
(3) strings that are used by the renderer -- for example, we imagine that all the things that are currently supplied on the CPU side as RendererServices methods, probably will need corresponding sections of Cuda code, supplied by the renderer, and the internals of those functions need to be able to know the unique values of those strings.

As further background, we should spell out that there are three sources of strings, really:
(1) "pre-known" strings that we know that are referenced inside the OSL runtime implementation, for example all the ustrings in the Strings:: namespace (examples: "RGB", "snoise", "wrap", etc.).
(2) strings that are specified in the shader as literals, or come from the scene as parameter instance values (such as most texture names), or are constructed in the shader in a way that can resolved during runtime optimization of the shader group. In other words, all the strings not in group 1 that the shader itself will try to reference.
(3) strings that are used by the renderer -- for example, we imagine that all the things that are currently supplied on the CPU side as RendererServices methods, probably will need corresponding sections of Cuda code, supplied by the renderer, and the internals of those functions need to be able to know the unique values of those strings.

I'm not quite convinced yet. Are those hash collisions @etheory mentioned for the current OIIO string hash? Were they generated to collide? Obviously this is possible since we don't use cryptographic hashes. The hope was that the odds of it happening by chance with real world strings would be low enough -- we're not making any claims that someone can't engineer strings to collide.
That being said - how bad would it be if we had to increase the size of the string type to 128 bits? At that point its definitely safe to ignore collisions.

I'm not quite convinced yet. Are those hash collisions @etheory mentioned for the current OIIO string hash? Were they generated to collide? Obviously this is possible since we don't use cryptographic hashes. The hope was that the odds of it happening by chance with real world strings would be low enough -- we're not making any claims that someone can't engineer strings to collide.
That being said - how bad would it be if we had to increase the size of the string type to 128 bits? At that point its definitely safe to ignore collisions.

@fpsunflower hi!
Yes they were engineered to collide ;-). I'm still working on getting something into production for a real-world example.
Increasing the number of bits won't matter.
The fact still stands. There is a non-zero change on a string collision for ANY length of hash.
IMO the only sensible thing to do is just know this, and ensure the system can handle hash collisions.
A few specific points to add to this thread:

The "pigeon-holing" that happens with hashes is not length-dependent, by design. It should be the same probability regardless of length. So the idea that longer strings or shorter strings are somehow more or less likely to collide, I think is misguided. Any decent hash function should ensure the probability of collision is effectively "independent" of string length.
Even though my example doesn't seem indicative, what about the case where facilities use asset-management systems that abstract filenames on disc, then return references to these files from some kind of data-base lookup. In that scenario, this kind of filename (the one I gave above in my counter-example) is 100% plausible, both in terms of length, and alphanumeric content. Not all facilities use "sensible-looking filenames" (caveat, AL does, but I've worked in facilities that don't).
Collisions simply do happen. I personally think building a system on the assumption that they won't, is not a good idea, and will at some point, lead to issues.

@fpsunflower hi!
Yes they were engineered to collide ;-). I'm still working on getting something into production for a real-world example.
Increasing the number of bits won't matter.
The fact still stands. There is a non-zero change on a string collision for ANY length of hash.
IMO the only sensible thing to do is just know this, and ensure the system can handle hash collisions.
A few specific points to add to this thread:

The "pigeon-holing" that happens with hashes is not length-dependent, by design. It should be the same probability regardless of length. So the idea that longer strings or shorter strings are somehow more or less likely to collide, I think is misguided. Any decent hash function should ensure the probability of collision is effectively "independent" of string length.
Even though my example doesn't seem indicative, what about the case where facilities use asset-management systems that abstract filenames on disc, then return references to these files from some kind of data-base lookup. In that scenario, this kind of filename (the one I gave above in my counter-example) is 100% plausible, both in terms of length, and alphanumeric content. Not all facilities use "sensible-looking filenames" (caveat, AL does, but I've worked in facilities that don't).
Collisions simply do happen. I personally think building a system on the assumption that they won't, is not a good idea, and will at some point, lead to issues.

* Collisions simply do happen. I personally think building a system on the assumption that they won't, is not a good idea, and will at some point, lead to issues.

I don't believe any of us were putting this forward as a permanent solution. We know that using the hash is playing with fire. We were just hoping that it would be rare enough to give us breathing room to come back to this issue later with a better solution.

* Collisions simply do happen. I personally think building a system on the assumption that they won't, is not a good idea, and will at some point, lead to issues.

I don't believe any of us were putting this forward as a permanent solution. We know that using the hash is playing with fire. We were just hoping that it would be rare enough to give us breathing room to come back to this issue later with a better solution.

Collision probability does depend on the hash length. If you use a 2 bit
hash you will have many collisions, not so much with 128bit. This frequent
enough discussion always reminds me this quote from Structure and
interpretation of computer programs (Abelson, Sussman), in a similar
context of primality testing:
"In testing primality of very large numbers chosen at random, the chance of
stumbling upon a value that fools the Fermat test is less than the chance
that cosmic radiation will cause the computer to make an error in carrying
out a “correct” algorithm. Considering an algorithm to be inadequate for
the first reason but not for the second illustrates the difference between
mathematics and engineering."
It is interesting that it's precisely the engineers losing sleep over this,
and not mathematicians :)
Like Chris said, one key question is: how bad is it if two strings collide
in the real world. On the probability of collision you also have to scale
by the probability of comparison.
On Tue, Jan 12, 2021 at 7:35 AM Larry Gritz<EMAIL_ADDRESS>wrote:

Collisions simply do happen. I personally think building a system on the assumption that they won't, is not a good idea, and will at some point, lead to issues.

I don't believe any of us were putting this forward as a permanent
solution. We know that using the hash is playing with fire. We were just
hoping that it would be rare enough to give us breathing room to come back
to this issue later with a better solution.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-758439557,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24H73WJMA6VQMQ42DDTSZPUSHANCNFSM4UOAKVPQ
.

Collision probability does depend on the hash length. If you use a 2 bit
hash you will have many collisions, not so much with 128bit. This frequent
enough discussion always reminds me this quote from Structure and
interpretation of computer programs (Abelson, Sussman), in a similar
context of primality testing:
"In testing primality of very large numbers chosen at random, the chance of
stumbling upon a value that fools the Fermat test is less than the chance
that cosmic radiation will cause the computer to make an error in carrying
out a “correct” algorithm. Considering an algorithm to be inadequate for
the first reason but not for the second illustrates the difference between
mathematics and engineering."
It is interesting that it's precisely the engineers losing sleep over this,
and not mathematicians :)
Like Chris said, one key question is: how bad is it if two strings collide
in the real world. On the probability of collision you also have to scale
by the probability of comparison.
On Tue, Jan 12, 2021 at 7:35 AM Larry Gritz<EMAIL_ADDRESS>wrote:

Collisions simply do happen. I personally think building a system on the assumption that they won't, is not a good idea, and will at some point, lead to issues.

I don't believe any of us were putting this forward as a permanent
solution. We know that using the hash is playing with fire. We were just
hoping that it would be rare enough to give us breathing room to come back
to this issue later with a better solution.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-758439557,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24H73WJMA6VQMQ42DDTSZPUSHANCNFSM4UOAKVPQ
.

We will have collisions, and we will have them on short, simple strings. That sounds counter-intuitive, but if you think about it, it MUST be that way.
Proof: The hash is 64 bits. EITHER (a) strings less than 8 chars (64 bits) have hash collisions against each other, or (b) the set of all <= 8 char strings perfectly hash to all possible 64 bit values with no collisions (that would be amazing). If (b) is the case, then they have filled the space of 64 bit values, so every string of >= 8 chars will be a hash collision, and it will collide with a string of <= 8 chars. So either way, you will find hash collisions copiously among even very short (< 8 chars) strings.
But memory is not big enough to have anywhere near 2^64 strings. A ustring table entry is 56 bytes (not counting the characters!), so filling even a huge workstation with 128GB RAM with nothing but the shortest ustrings possible will max out memory at only a couple billion strings.
The danger is that all the usual analysis assumes random strings, but that's not what we're dealing with. The insidious cases would be things like two texture names
/path/to/texture/A/B/C/diffuse.exr
/path/to/texture/C/B/A/diffuse.exr

producing the same hash value. The hash function may be pretty good for random input, or even "sequential" input, but for certain structured inputs may be sensitive to permutations that crop up typically in a production data set.

We will have collisions, and we will have them on short, simple strings. That sounds counter-intuitive, but if you think about it, it MUST be that way.
Proof: The hash is 64 bits. EITHER (a) strings less than 8 chars (64 bits) have hash collisions against each other, or (b) the set of all <= 8 char strings perfectly hash to all possible 64 bit values with no collisions (that would be amazing). If (b) is the case, then they have filled the space of 64 bit values, so every string of >= 8 chars will be a hash collision, and it will collide with a string of <= 8 chars. So either way, you will find hash collisions copiously among even very short (< 8 chars) strings.
But memory is not big enough to have anywhere near 2^64 strings. A ustring table entry is 56 bytes (not counting the characters!), so filling even a huge workstation with 128GB RAM with nothing but the shortest ustrings possible will max out memory at only a couple billion strings.
The danger is that all the usual analysis assumes random strings, but that's not what we're dealing with. The insidious cases would be things like two texture names
/path/to/texture/A/B/C/diffuse.exr
/path/to/texture/C/B/A/diffuse.exr

producing the same hash value. The hash function may be pretty good for random input, or even "sequential" input, but for certain structured inputs may be sensitive to permutations that crop up typically in a production data set.

There's a collision hazard even in the current implementation, since the name of the CUDA variable is based on the hash (see the cute TODO in getOrAllocateCUDAVariable(). I guess kicking the can down the road has finally developed into street hockey. 😃
It should be possible to prove that there are no collisions in the set of static strings. The path-like strings are still problematic. At least for texture paths, isn't it the case that those must be known in advance and resolved to a texture handle? Are there other cases of string-specified resources we need to worry about? Named transforms and such?
If all strings were "laundered" through ustring, would it be possible to detect and warn about any collisions?

There's a collision hazard even in the current implementation, since the name of the CUDA variable is based on the hash (see the cute TODO in getOrAllocateCUDAVariable(). I guess kicking the can down the road has finally developed into street hockey. 😃
It should be possible to prove that there are no collisions in the set of static strings. The path-like strings are still problematic. At least for texture paths, isn't it the case that those must be known in advance and resolved to a texture handle? Are there other cases of string-specified resources we need to worry about? Named transforms and such?
If all strings were "laundered" through ustring, would it be possible to detect and warn about any collisions?

Oh, that's an interesting observation. If we have the same hash problem prior to this patch, then there's no reason not to merge it, right? Even if we keep it on our list to find a more robust long-term solution to this problem.

Oh, that's an interesting observation. If we have the same hash problem prior to this patch, then there's no reason not to merge it, right? Even if we keep it on our list to find a more robust long-term solution to this problem.

@timgrant (asked in a constructive manner)  So if we could "detect" the collisions, what would user do about it?  Start changing filenames or transform names until the warning goes away?

@timgrant (asked in a constructive manner)  So if we could "detect" the collisions, what would user do about it?  Start changing filenames or transform names until the warning goes away?

Good question, Alex. I wasn't thinking of it as a solution, but more as a very rough indicator of how well (or poorly) the hashing approach is working. But it's maybe a bit like those "Days Since Last Accident" signs. It's the next accident we need to worry about, and it's anyone's guess how minor or severe it might be. I guess it might be useful for our initial testing, but probably isn't something a user would want to rely on.

Good question, Alex. I wasn't thinking of it as a solution, but more as a very rough indicator of how well (or poorly) the hashing approach is working. But it's maybe a bit like those "Days Since Last Accident" signs. It's the next accident we need to worry about, and it's anyone's guess how minor or severe it might be. I guess it might be useful for our initial testing, but probably isn't something a user would want to rely on.

OK, how about this crazy idea:
The ustring class can keep a second hash table internally -- a reverse map that maps hash values to ustrings. (Let's call this the ID map.)
When a new ustring table entry is created for the first time by make_unique(s), we start with h = hash(s) and nrh = 0.  (nrh stands for "number of re-hashes"). The IDmap[h] is checked, if there is no entry for that ID, it's added to the ID table as well as doing usual insertion into the ustring table, and we're done.  If there is already an entry for that ID and it's this very string, also we're done.
But if there is different string with that ID, we rehash (for example, h = hash(h, ++nrh)), and repeat the process until we either find our string or else get to an empty ID table entry, i.e. we have a string we've never seen before and a unique h value, and that becomes the hash of this new ustring.
I claim that this has a few consequences:

We'll never have two ustrings with identical final hash values, so the GPU side using the hash as a proxy for the ustring will not lead to any errors.

For the vast, vast majority of scenes that we will ever encounter, there will be no collisions, no strings that get rehashed, all strings will have precisely the same hash as every other frame we rendered, and so the PTX cache works wonderfully.

Every once in a great while (and for any particular user, shot, or production, very possibly never), there will be ustring collisions, and some string that is referenced in a shader will end up with a different hash value than it was assigned for the other frames we have previously rendered, and so the PTX cache will not be a match and the shader will be correct, but will take longer than usual to compile. (Though, the PTX cache will be helpful the next time we render that frame, or for that matter, any other frame that has the same set of collision resolutions.)

Costs: There is no extra cost for using constructed ustrings (=, ==, !=). There is a slight additional cost for constructing ustrings from characters -- checking or amending the ID table, and very very occasionally having to do the rehashing and multiple probes. And there is additional memory use to store the ID table itself (another 16 bytes or so per unique ustring, compared to I think 56 bytes currently for the internal table entry, so maybe the internal non-character, table-related storage behind the ustring implementation will increase by around 25%).
A couple of minor implementation notes:
It might be kind to remove the extra expenses of all this from the non-OSL users of ustring (and even OSL doesn't need it when not being used by a GPU renderer), by having runtime-settable switch that tells ustring implementation whether to maintain the ID table.
We may or may not wish to care that the implementation of OSL's hash(string) function may -- very very occasionally -- not be 100% stable from scene to scene. I don't know if anybody uses OSL's hash function in such a way that they rely on its absolute stability across all possible scenes. But if this is important, we can probably figure something out, like storing in the internal ustring table both the "original" and "final" hashes, and OSL's hash() returns the original one. (Yeah, yeah, how do we get that on the GPU side? Maybe, like the contents of the characters, we just don't... for now.)

OK, how about this crazy idea:
The ustring class can keep a second hash table internally -- a reverse map that maps hash values to ustrings. (Let's call this the ID map.)
When a new ustring table entry is created for the first time by make_unique(s), we start with h = hash(s) and nrh = 0.  (nrh stands for "number of re-hashes"). The IDmap[h] is checked, if there is no entry for that ID, it's added to the ID table as well as doing usual insertion into the ustring table, and we're done.  If there is already an entry for that ID and it's this very string, also we're done.
But if there is different string with that ID, we rehash (for example, h = hash(h, ++nrh)), and repeat the process until we either find our string or else get to an empty ID table entry, i.e. we have a string we've never seen before and a unique h value, and that becomes the hash of this new ustring.
I claim that this has a few consequences:

We'll never have two ustrings with identical final hash values, so the GPU side using the hash as a proxy for the ustring will not lead to any errors.

For the vast, vast majority of scenes that we will ever encounter, there will be no collisions, no strings that get rehashed, all strings will have precisely the same hash as every other frame we rendered, and so the PTX cache works wonderfully.

Every once in a great while (and for any particular user, shot, or production, very possibly never), there will be ustring collisions, and some string that is referenced in a shader will end up with a different hash value than it was assigned for the other frames we have previously rendered, and so the PTX cache will not be a match and the shader will be correct, but will take longer than usual to compile. (Though, the PTX cache will be helpful the next time we render that frame, or for that matter, any other frame that has the same set of collision resolutions.)

Costs: There is no extra cost for using constructed ustrings (=, ==, !=). There is a slight additional cost for constructing ustrings from characters -- checking or amending the ID table, and very very occasionally having to do the rehashing and multiple probes. And there is additional memory use to store the ID table itself (another 16 bytes or so per unique ustring, compared to I think 56 bytes currently for the internal table entry, so maybe the internal non-character, table-related storage behind the ustring implementation will increase by around 25%).
A couple of minor implementation notes:
It might be kind to remove the extra expenses of all this from the non-OSL users of ustring (and even OSL doesn't need it when not being used by a GPU renderer), by having runtime-settable switch that tells ustring implementation whether to maintain the ID table.
We may or may not wish to care that the implementation of OSL's hash(string) function may -- very very occasionally -- not be 100% stable from scene to scene. I don't know if anybody uses OSL's hash function in such a way that they rely on its absolute stability across all possible scenes. But if this is important, we can probably figure something out, like storing in the internal ustring table both the "original" and "final" hashes, and OSL's hash() returns the original one. (Yeah, yeah, how do we get that on the GPU side? Maybe, like the contents of the characters, we just don't... for now.)

The fact that there are collisions doesn’t mean you’re going to find one.
And structured input shouldn’t be an issue if the hash is good. A hash is
usually engineered so if you change a bit of the input, about half the bits
of the output change.
I suggest we put the alarms for collisions so we sleep better. Just for the
problem to not happen silently.
An emergency fix if it happens could be to have a global salt string for
the hash. If collisions are detected you can change the salt instead of
changing file paths.
On Wed, 13 Jan 2021 at 06:43, timgrant<EMAIL_ADDRESS>wrote:

Good question, Alex. I wasn't thinking of it as a solution, but more as a
very rough indicator of how well (or poorly) the hashing approach is
working. But it's maybe a bit like those "Days Since Last Accident" signs.
It's the next accident we need to worry about, and it's anyone's guess how
minor or severe it might be. I guess it might be useful for our initial
testing, but probably isn't something a user would want to rely on.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-759219444,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24GK3M22TLEDROHDLSDSZUXGTANCNFSM4UOAKVPQ
.

The fact that there are collisions doesn’t mean you’re going to find one.
And structured input shouldn’t be an issue if the hash is good. A hash is
usually engineered so if you change a bit of the input, about half the bits
of the output change.
I suggest we put the alarms for collisions so we sleep better. Just for the
problem to not happen silently.
An emergency fix if it happens could be to have a global salt string for
the hash. If collisions are detected you can change the salt instead of
changing file paths.
On Wed, 13 Jan 2021 at 06:43, timgrant<EMAIL_ADDRESS>wrote:

Good question, Alex. I wasn't thinking of it as a solution, but more as a
very rough indicator of how well (or poorly) the hashing approach is
working. But it's maybe a bit like those "Days Since Last Accident" signs.
It's the next accident we need to worry about, and it's anyone's guess how
minor or severe it might be. I guess it might be useful for our initial
testing, but probably isn't something a user would want to rely on.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-759219444,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24GK3M22TLEDROHDLSDSZUXGTANCNFSM4UOAKVPQ
.

Rehashing is appealing, but it means it can no longer be a constexpr. Don’t
know how much Cliff will like the idea.
On Wed, 13 Jan 2021 at 07:48, Larry Gritz<EMAIL_ADDRESS>wrote:

OK, how about this crazy idea:
The ustring class can keep a second hash table internally -- a reverse map
that maps hash values to ustrings. (Let's call this the ID map.)
When a new ustring table entry is created for the first time by
make_unique(s), we start with h = hash(s) and nrh = 0. (nrh stands for
"number of re-hashes"). The IDmap[h] is checked, if there is no entry for
that ID, it's added to the ID table as well as doing usual insertion into
the ustring table, and we're done. If there is already an entry for that ID
and it's this very string, also we're done.
But if there is different string with that ID, we rehash (for example, h
= hash(h, ++nrh)), and repeat the process until we either find our string
or else get to an empty ID table entry, i.e. we have a string we've never
seen before and a unique h value, and that becomes the hash of this new
ustring.
I claim that this has a few consequences:

We'll never have two ustrings with identical final hash values, so the
GPU side using the hash as a proxy for the ustring will not lead to any
errors.
2.
For the vast, vast majority of scenes that we will ever encounter,
there will be no collisions, no strings that get rehashed, all strings will
have precisely the same hash as every other frame we rendered, and so the
PTX cache works wonderfully.
3.
Every once in a great while (and for any particular user, shot, or
production, very possibly never), there will be ustring collisions, and
some string that is referenced in a shader will end up with a different
hash value than it was assigned for the other frames we have previously
rendered, and so the PTX cache will not be a match and the shader will be
correct, but will take longer than usual to compile. (Though, the
PTX cache will be helpful the next time we render that frame, or for
that matter, any other frame that has the same set of collision
resolutions.)
Costs: There is no extra cost for using constructed ustrings (=, ==, !=).
There is a slight additional cost for constructing ustrings from characters
-- checking or amending the ID table, and very very occasionally having to
do the rehashing and multiple probes. And there is additional memory use to
store the ID table itself (another 16 bytes or so per unique ustring,
compared to I think 56 bytes currently for the internal table entry, so
maybe the internal non-character, table-related storage behind the ustring
implementation will increase by around 25%).
A couple of minor implementation notes:
It might be kind to remove the extra expenses of all this from the non-OSL
users of ustring (and even OSL doesn't need it when not being used by a GPU
renderer), by having runtime-settable switch that tells ustring
implementation whether to maintain the ID table.
We may or may not wish to care that the implementation of OSL's
hash(string) function may -- very very occasionally -- not be 100% stable
from scene to scene. I don't know if anybody uses OSL's hash function in
such a way that they rely on its absolute stability across all possible
scenes. But if this is important, we can probably figure something out,
like storing in the internal ustring table both the "original" and "final"
hashes, and OSL's hash() returns the original one. (Yeah, yeah, how do we
get that on the GPU side? Maybe, like the contents of the characters, we
just don't... for now.)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-759244312,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24B5RUNRW6EU2RZZLPTSZU633ANCNFSM4UOAKVPQ
.

Rehashing is appealing, but it means it can no longer be a constexpr. Don’t
know how much Cliff will like the idea.
On Wed, 13 Jan 2021 at 07:48, Larry Gritz<EMAIL_ADDRESS>wrote:

OK, how about this crazy idea:
The ustring class can keep a second hash table internally -- a reverse map
that maps hash values to ustrings. (Let's call this the ID map.)
When a new ustring table entry is created for the first time by
make_unique(s), we start with h = hash(s) and nrh = 0. (nrh stands for
"number of re-hashes"). The IDmap[h] is checked, if there is no entry for
that ID, it's added to the ID table as well as doing usual insertion into
the ustring table, and we're done. If there is already an entry for that ID
and it's this very string, also we're done.
But if there is different string with that ID, we rehash (for example, h
= hash(h, ++nrh)), and repeat the process until we either find our string
or else get to an empty ID table entry, i.e. we have a string we've never
seen before and a unique h value, and that becomes the hash of this new
ustring.
I claim that this has a few consequences:

We'll never have two ustrings with identical final hash values, so the
GPU side using the hash as a proxy for the ustring will not lead to any
errors.
2.
For the vast, vast majority of scenes that we will ever encounter,
there will be no collisions, no strings that get rehashed, all strings will
have precisely the same hash as every other frame we rendered, and so the
PTX cache works wonderfully.
3.
Every once in a great while (and for any particular user, shot, or
production, very possibly never), there will be ustring collisions, and
some string that is referenced in a shader will end up with a different
hash value than it was assigned for the other frames we have previously
rendered, and so the PTX cache will not be a match and the shader will be
correct, but will take longer than usual to compile. (Though, the
PTX cache will be helpful the next time we render that frame, or for
that matter, any other frame that has the same set of collision
resolutions.)
Costs: There is no extra cost for using constructed ustrings (=, ==, !=).
There is a slight additional cost for constructing ustrings from characters
-- checking or amending the ID table, and very very occasionally having to
do the rehashing and multiple probes. And there is additional memory use to
store the ID table itself (another 16 bytes or so per unique ustring,
compared to I think 56 bytes currently for the internal table entry, so
maybe the internal non-character, table-related storage behind the ustring
implementation will increase by around 25%).
A couple of minor implementation notes:
It might be kind to remove the extra expenses of all this from the non-OSL
users of ustring (and even OSL doesn't need it when not being used by a GPU
renderer), by having runtime-settable switch that tells ustring
implementation whether to maintain the ID table.
We may or may not wish to care that the implementation of OSL's
hash(string) function may -- very very occasionally -- not be 100% stable
from scene to scene. I don't know if anybody uses OSL's hash function in
such a way that they rely on its absolute stability across all possible
scenes. But if this is important, we can probably figure something out,
like storing in the internal ustring table both the "original" and "final"
hashes, and OSL's hash() returns the original one. (Yeah, yeah, how do we
get that on the GPU side? Maybe, like the contents of the characters, we
just don't... for now.)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-759244312,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24B5RUNRW6EU2RZZLPTSZU633ANCNFSM4UOAKVPQ
.

Aagh, dammit. I forgot about the constexpr. How badly does that hurt us?

Aagh, dammit. I forgot about the constexpr. How badly does that hurt us?

So constexpr is important because that's how compiled Cuda code gets the hash value representing the string from the characters. It needs to be constexpr because it has no access to the CPU-side internal ustring tables.
But bear with me... it doesn't matter.
Any compiled Cuda code (or liboslexec-side and renderer-side C++ code) will have strings that are (a) few, (b) fixed. All the "dynamic" strings -- meaning it could be different from run to run and possibly a large number of unanticipated strings -- will come into the system via the shaders themselves and in particular the parameters (such as texture names).
So the definition of the chars-to-ustring constructor on the Cuda side is: the old constexpr hash, just like in this PR.
Remember, that's always the first guess for the hash, and almost always the final hash, except in the rare, rare case that there is a collision. So we assume (and we can check to verify) that the small number of ustrings statically constructed in the renderer and the OSL library do not produce a collision against each other. If they do, we're in trouble, but the chances of that are minuscule for a small number of strings and a 64 bit hash space. We can have code that verifies that it's collision-free before we load the shaders.
Only the CPU side, subsequently constructed, strings have a chance at colliding, so it's only there that we use the non-constexpr, collision-handling constructor. Those strings just have their final hash values passed to the Cuda code, they don't need to be constructed in the Cuda code.

So constexpr is important because that's how compiled Cuda code gets the hash value representing the string from the characters. It needs to be constexpr because it has no access to the CPU-side internal ustring tables.
But bear with me... it doesn't matter.
Any compiled Cuda code (or liboslexec-side and renderer-side C++ code) will have strings that are (a) few, (b) fixed. All the "dynamic" strings -- meaning it could be different from run to run and possibly a large number of unanticipated strings -- will come into the system via the shaders themselves and in particular the parameters (such as texture names).
So the definition of the chars-to-ustring constructor on the Cuda side is: the old constexpr hash, just like in this PR.
Remember, that's always the first guess for the hash, and almost always the final hash, except in the rare, rare case that there is a collision. So we assume (and we can check to verify) that the small number of ustrings statically constructed in the renderer and the OSL library do not produce a collision against each other. If they do, we're in trouble, but the chances of that are minuscule for a small number of strings and a 64 bit hash space. We can have code that verifies that it's collision-free before we load the shaders.
Only the CPU side, subsequently constructed, strings have a chance at colliding, so it's only there that we use the non-constexpr, collision-handling constructor. Those strings just have their final hash values passed to the Cuda code, they don't need to be constructed in the Cuda code.

How do we check the cuda side hashes? Aren't the chars lost at compile time?
On Wed, Jan 13, 2021 at 7:12 PM Larry Gritz<EMAIL_ADDRESS>wrote:

So constexpr is important because that's how compiled Cuda code gets the
hash value representing the string from the characters. It needs to be
constexpr because it has no access to the CPU-side internal ustring tables.
But bear with me... it doesn't matter.
Any compiled Cuda code (or liboslexec-side and renderer-side C++ code)
will have strings that are (a) few, (b) fixed. All the "dynamic" strings --
meaning it could be different from run to run and possibly a large number
of unanticipated strings -- will come into the system via the shaders
themselves and in particular the parameters (such as texture names).
So the definition of the chars-to-ustring constructor on the Cuda side is:
the old constexpr hash, just like in this PR.
Remember, that's always the first guess for the hash, and almost always
the final hash, except in the rare, rare case that there is a collision. So
we assume (and we can check to verify) that the small number of ustrings
statically constructed in the renderer and the OSL library do not produce a
collision against each other. If they do, we're in trouble, but the chances
of that are minuscule for a small number of strings and a 64 bit hash
space. We can have code that verifies that it's collision-free before we
load the shaders.
Only the CPU side, subsequently constructed, strings have a chance at
colliding, so it's only there that we use the non-constexpr,
collision-handling constructor. Those strings just have their final hash
values passed to the Cuda code, they don't need to be constructed in the
Cuda code.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-759626433,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24ATBGTASQ2TTOJPRZTSZXPAXANCNFSM4UOAKVPQ
.

How do we check the cuda side hashes? Aren't the chars lost at compile time?
On Wed, Jan 13, 2021 at 7:12 PM Larry Gritz<EMAIL_ADDRESS>wrote:

So constexpr is important because that's how compiled Cuda code gets the
hash value representing the string from the characters. It needs to be
constexpr because it has no access to the CPU-side internal ustring tables.
But bear with me... it doesn't matter.
Any compiled Cuda code (or liboslexec-side and renderer-side C++ code)
will have strings that are (a) few, (b) fixed. All the "dynamic" strings --
meaning it could be different from run to run and possibly a large number
of unanticipated strings -- will come into the system via the shaders
themselves and in particular the parameters (such as texture names).
So the definition of the chars-to-ustring constructor on the Cuda side is:
the old constexpr hash, just like in this PR.
Remember, that's always the first guess for the hash, and almost always
the final hash, except in the rare, rare case that there is a collision. So
we assume (and we can check to verify) that the small number of ustrings
statically constructed in the renderer and the OSL library do not produce a
collision against each other. If they do, we're in trouble, but the chances
of that are minuscule for a small number of strings and a 64 bit hash
space. We can have code that verifies that it's collision-free before we
load the shaders.
Only the CPU side, subsequently constructed, strings have a chance at
colliding, so it's only there that we use the non-constexpr,
collision-handling constructor. Those strings just have their final hash
values passed to the Cuda code, they don't need to be constructed in the
Cuda code.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-759626433,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24ATBGTASQ2TTOJPRZTSZXPAXANCNFSM4UOAKVPQ
.

The rule is: If you have a constructed ustring from a string literal on the Cuda side, you need a statically initialized ustring of the same string in your C++ side code. That ensures that the string is constructed early in the render, before any of the dynamic strings have been constructed.
To verify, OSL asks ustring if there were any collisions so far, fairly early in its operations (before any of the shaders have been loaded, so only the statically constructed ones are present). They you're home-free.

The rule is: If you have a constructed ustring from a string literal on the Cuda side, you need a statically initialized ustring of the same string in your C++ side code. That ensures that the string is constructed early in the render, before any of the dynamic strings have been constructed.
To verify, OSL asks ustring if there were any collisions so far, fairly early in its operations (before any of the shaders have been loaded, so only the statically constructed ones are present). They you're home-free.

So in Cuda code, you can't say
if (mysr == devustring("blah")) ...

but rather, (in essence, I know the syntax isn't right)
OSL_HOSTDEVICE static devustring ublah("blah");
if (mystr == ublah) ...

devustring is identical to ustring on the CPU side, but on the Cuda side its constructor from a string literal just computes the constexpr 64 bit hash. The HOSTDEVICE and static add up to the fact that the C++/CPU side will make the "correct" ustring with the same hash early enough that there will not be any collisions until later on when all the textures and what not are added.

So in Cuda code, you can't say
if (mysr == devustring("blah")) ...

but rather, (in essence, I know the syntax isn't right)
OSL_HOSTDEVICE static devustring ublah("blah");
if (mystr == ublah) ...

devustring is identical to ustring on the CPU side, but on the Cuda side its constructor from a string literal just computes the constexpr 64 bit hash. The HOSTDEVICE and static add up to the fact that the C++/CPU side will make the "correct" ustring with the same hash early enough that there will not be any collisions until later on when all the textures and what not are added.

@lgritz Larry, I like the rehash approach combined with the strdecls.h STRING_PARAM to seed the CPU side reverse look with the 1st hash.  Simple rule to never use devstring natively on CUDA side, just always use STRING_PARAM (which I think this existing PR does?).  Perhaps there could be a version of the constructor that declares the result must be the 1st hash.
So as long as all STRING_PARAMS don't collide with anything that already exists on the CPU side after a renderer is loaded.
NOTE: we may not be in control of what other 3rd party (aka the renderer) code has populated the ustring table with before our static constructors happen.  However we can certainly detect a collison for these STRING_PARAMS and any existing ustrings (with different string contents) and throw an exception.  Alternatively we could detect it, maybe report it, but NOT do the rehash just use the 1st hash value as the "final" hash that way at least.  We are back to having a "potential" incorrect equality comparison but only if those 2 ustring values share the same hash code are compared.
And as Larry outlined, all dynamic JIT'd strings would use the rehashed "final" hash code which handles collisions.  The only downside being the resulting final hash code "could" be different run to run depending on order which ustrings are registered in the lookup table.  You still get "correct" results, just the shader's JIT'd source would have different final hash code values harded in it (defeating caching in drivers).  However this is just to handle collisions.
Seems like a decent trade off vs. incorrect results.

@lgritz Larry, I like the rehash approach combined with the strdecls.h STRING_PARAM to seed the CPU side reverse look with the 1st hash.  Simple rule to never use devstring natively on CUDA side, just always use STRING_PARAM (which I think this existing PR does?).  Perhaps there could be a version of the constructor that declares the result must be the 1st hash.
So as long as all STRING_PARAMS don't collide with anything that already exists on the CPU side after a renderer is loaded.
NOTE: we may not be in control of what other 3rd party (aka the renderer) code has populated the ustring table with before our static constructors happen.  However we can certainly detect a collison for these STRING_PARAMS and any existing ustrings (with different string contents) and throw an exception.  Alternatively we could detect it, maybe report it, but NOT do the rehash just use the 1st hash value as the "final" hash that way at least.  We are back to having a "potential" incorrect equality comparison but only if those 2 ustring values share the same hash code are compared.
And as Larry outlined, all dynamic JIT'd strings would use the rehashed "final" hash code which handles collisions.  The only downside being the resulting final hash code "could" be different run to run depending on order which ustrings are registered in the lookup table.  You still get "correct" results, just the shader's JIT'd source would have different final hash code values harded in it (defeating caching in drivers).  However this is just to handle collisions.
Seems like a decent trade off vs. incorrect results.

The other nice thing is that this idea builds on Cliff's PR here, it doesn't replace it. All it does is ensure no collisions for subsequently added ustrings. If others agree that this is the right approach, we can commit this PR now, and then add the collision-fixing layer at our leisure. It doesn't hold up any other progress.

The other nice thing is that this idea builds on Cliff's PR here, it doesn't replace it. All it does is ensure no collisions for subsequently added ustrings. If others agree that this is the right approach, we can commit this PR now, and then add the collision-fixing layer at our leisure. It doesn't hold up any other progress.

formatting now clang-format friendly

formatting now clang-format friendly

We decided at the TSC meeting that we don't want to merge this quite yet, because of the hash collision problem. My proposal above was accepted, so I'm going to implement it (much of the work is on the OIIO side), and we'll merge this when my complement to it is ready to also be tried.

We decided at the TSC meeting that we don't want to merge this quite yet, because of the hash collision problem. My proposal above was accepted, so I'm going to implement it (much of the work is on the OIIO side), and we'll merge this when my complement to it is ready to also be tried.

I did the test
$ find /shots/show1 | tee /tmp/strings.txt | wc
I got bored when it got to 800k strings and stopped it. Then run the
farmhash 64bit on all those strings.
No collisions
That was an animated show. Now trying a vfx show, and only regular files:
$ find -L /shots/show2 -type f | tee /tmp/strings.txt | wc
Got tired of waiting at 2.5 million strings. Again, no collisions
Now for the math, which is exactly the same as the birthday problem
https://en.wikipedia.org/wiki/Birthday_problem
The probability of one or more collisions can be approximated by 1 -
e^(-n^2 / (2 m)) where n is the number of strings and m the hash space.
If farmhash were perfect the hash space would be 2^64, and the probability
of collision in 800k strings would be around 10e-8, 10e-7 for 2.5M. Let's
suppose that farmhash is 10bits away from being perfect (so pretty bad),
the hash space is 2^54 and the probability goes up to 10e-5, more or less
like winning the lottery.
Now take into account no frame is going to reference 800k - 2.5M strings,
and if you factor in the probability of comparison (on the GPU), the
probability of failure is just ridiculous. With a 32bit hash on 800k
strings, even if perfect, the probability of collision is almost 1.0, so
you have a problem there. But with 64bit it is very rare. Collisions do
exist, of course. There also exists a number that when squared gives you 2
:) But finding one in our small world is a different story.
Just wanted to run the numbers. I know you guys are 100% convinced the
simple hash is not acceptable. Or is it 99.999999% convinced? ;)
On Fri, Jan 22, 2021 at 1:08 AM Larry Gritz<EMAIL_ADDRESS>wrote:

We decided at the TSC meeting that we don't want to merge this quite yet,
because of the hash collision problem. My proposal above was accepted, so
I'm going to implement it (much of the work is on the OIIO side), and we'll
merge this when my complement to it is ready to also be tried.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-765021305,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24H4TKQTZCPGYUTEQ6LS3C6WNANCNFSM4UOAKVPQ
.

I did the test
$ find /shots/show1 | tee /tmp/strings.txt | wc
I got bored when it got to 800k strings and stopped it. Then run the
farmhash 64bit on all those strings.
No collisions
That was an animated show. Now trying a vfx show, and only regular files:
$ find -L /shots/show2 -type f | tee /tmp/strings.txt | wc
Got tired of waiting at 2.5 million strings. Again, no collisions
Now for the math, which is exactly the same as the birthday problem
https://en.wikipedia.org/wiki/Birthday_problem
The probability of one or more collisions can be approximated by 1 -
e^(-n^2 / (2 m)) where n is the number of strings and m the hash space.
If farmhash were perfect the hash space would be 2^64, and the probability
of collision in 800k strings would be around 10e-8, 10e-7 for 2.5M. Let's
suppose that farmhash is 10bits away from being perfect (so pretty bad),
the hash space is 2^54 and the probability goes up to 10e-5, more or less
like winning the lottery.
Now take into account no frame is going to reference 800k - 2.5M strings,
and if you factor in the probability of comparison (on the GPU), the
probability of failure is just ridiculous. With a 32bit hash on 800k
strings, even if perfect, the probability of collision is almost 1.0, so
you have a problem there. But with 64bit it is very rare. Collisions do
exist, of course. There also exists a number that when squared gives you 2
:) But finding one in our small world is a different story.
Just wanted to run the numbers. I know you guys are 100% convinced the
simple hash is not acceptable. Or is it 99.999999% convinced? ;)
On Fri, Jan 22, 2021 at 1:08 AM Larry Gritz<EMAIL_ADDRESS>wrote:

We decided at the TSC meeting that we don't want to merge this quite yet,
because of the hash collision problem. My proposal above was accepted, so
I'm going to implement it (much of the work is on the OIIO side), and we'll
merge this when my complement to it is ready to also be tried.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/imageworks/OpenShadingLanguage/pull/1309#issuecomment-765021305,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADJ24H4TKQTZCPGYUTEQ6LS3C6WNANCNFSM4UOAKVPQ
.

Updated a few things:

cleaned up string_hash.h to use the "constexpr" hash that now resides in OIIO
rebased this on the top of trunk so things don't get too out of date.

Hey, Cliff, can you add a DCO signoff to the latest commit, just
git commit --amend -s

ought to do it, then re-push (with force).

This has waited far too long. I'm going to merge this today if nobody has a strong objection. We're very close to being ready to merge the OIIO side that makes hash collisions impossible. If you are super concerned about the extremely low chance of collisions, then, just don't update to this commit on OSL until we've committed the OIIO side.


Combined Smoothing Floats and Local/Remote Fix

Addition of smoother name to combine smoother floats to reduce parameters need in the animator.
Creates Local and Remote version of the parameter to control to the smoothing.
Moved generated animations to OSCmooth\Generated\Anim folder. Should keep custom smoothers separate

Known issues with the UI

first instance of the Smoother name will be precedence for local and remote values any smoothers with the same name.
No overwriting of existing parameters values i.e changing the smoothness value will not change

I have made changes fixes nearly all of these issues, the tool was not ready for deployment quite yet.
Would recommend rebasing to my latest master since there are a huge amount of changes to many of the systems developed to better accommodate for future updates. I guess I version controlled too early lol.


As a Marist Student, I can report others so that I can limit harmful behavior.

Low Priority... button that can be put on the profile page that will allow the user to report the driver for any suspicious behavior.


Remove Phoenix POST Signup

What's this PR do?
Removes code for posting a Signup to the Phoenix API.
How should this be reviewed?
Send keyword for User who hasn't signed up and confirm signup is created.
Relevant tickets
#974
Checklist

[ ] Tested on staging.

Codecov Report

Merging #975 into develop will increase coverage by 0.36%.
The diff coverage is 0%.

@@             Coverage Diff             @@
##           develop     #975      +/-   ##
===========================================
+ Coverage    58.02%   58.38%   +0.36%
===========================================
  Files           70       70
  Lines         1589     1574      -15
===========================================
- Hits           922      919       -3
+ Misses         667      655      -12

Impacted Files
Coverage Δ

lib/rogue.js
30.76% <ø> (-2.57%)
:arrow_down:

config/lib/rogue.js
100% <ø> (ø)
:arrow_up:

lib/phoenix.js
50% <ø> (+11.53%)
:arrow_up:

app/models/Signup.js
13% <0%> (+0.3%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c979cf4...b65b9a9. Read the comment docs.


[Feature] Adding Magnetic force on a moving charge Calculator

Description
It is an physics related calculator used to find the force on a moving charge.
Inputs
Formula:
𝐹=𝑞𝑣𝐵sin(𝜃)
Inputs: Charge (q), Velocity (v), Magnetic field strength (B), Angle between velocity and magnetic field (𝜃)
Output: Magnetic force (F)
It is very useful for the physics calculations in magnetic field chanpters
@Rakesh9100 sir, I would be happy to contribute to the project. Could you please assign me this issue so that I can work on it?
Thank you
Screenshots

Checklist

[X] I have checked the existing issues.
[X] I follow Contributing Guidelines of this project.
[X] I am a GSSOC'24 contributor.
[ ] I am a IWOC'24 contributor.
[ ] I am a JWOC'24 contributor.
[ ] I am a SWOC'24 contributor.

Code of Conduct

[X] I agree to follow this project's "Code of Conduct".

@Rakesh9100 this issue can be assigned

What will be the name of your calculator in short and specific @GAVINESHWAR

@Rakesh9100 Lorentz Force calculator sir ,

@Rakesh9100 assign me this issue to make contribution sir.
Thank you

Pls go ahead @GAVINESHWAR


Question about connection to game

Hi all and devs especially.
I am data-engineer and want to make dataset of survivio. To do this, I want to record some games. Recording screencasts and transcoding them is too difficult so I wanted to try direct connection to game.
Mainly, I do request to http://surviv.io/api/find_game. And get json like this:
 {"res":[{"zone":"waw","gameId":"53602255248287e3ec89cd00f9e4cda0","hosts":["eu-waw-01.surviv.io:449","eu-waw-01.surviv.io"],"addrs":["<IP_ADDRESS>:444","<IP_ADDRESS>"],"data":"ksErP4tovbwUtVpO4iSF4guHzJa7CKjHFqfPGENhih0elF8W9qZFHwIeIdvKsErDTg7iMqFurN+McUXW49OyPwi2LBH0xkWZeFyJudGCnKE8WwoFtjKQRmULb22mCnKi9ZfQPNwMWapYCiNxRvDUts1YriBe93Sgm0WV0CzXC5Fn+H4uNzP4WaFjR3JlgqolbdprgrXeyuGgC7tZ5FO4OrJDd/u96p/75G8dUSUCB1tffZ3OgLdXZ7LNpMe3eVpNssG0BDPJGItu","useHttps":false}]}

Data from this json is used to connect to survivio server via websocket protocol.
But I can't find how to make this data to websocket request. For aforementioned json my browser creates this request (coded as curl):
curl 'ws://<IP_ADDRESS>:444/play?gameId=53602255248287e3ec89cd00f9e4cda0' -H 'Pragma: no-cache' -H 'Origin: http://surviv.io' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7' -H 'Sec-WebSocket-Key: +qLjgICDV4BwAw9UnVJEWQ==' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36' -H 'Upgrade: websocket' -H 'Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits' -H 'Cache-Control: no-cache' -H 'Connection: Upgrade' -H 'Sec-WebSocket-Version: 13' --compressed

How to cast Sec-WebSocketKey from 'data' field in json? Any ideas or advice?
I've read that this Key is established during client-server handshake procedure.
But when I tried using websocket-client library to make direct connection to server with known game id (that way handshake will be completely new as server should count me as another new player), connection was established and stopped after several packets. I guess in json some additional data is encoded, but don't know how to interpret it.
I checked whole repository, but as python developer I am not good in js and no direct answer was found.
As another option I guess one can modify cheat source code so all incomin messages would be logged to file. This is also would be also fine to me. Could anyone show me which lines in app should be modified?
I appreciate any help. thanks in advance

Ok, I will summon Kalaborative

If you mean modifying the game's source code, see background.js: you can search the source code and replace with a console.log using regex
I'm unfamiliar with how to mock websocket requests so I'm not sure how to help there... my guess is the server is attaching an encoded session key before making the request, but I don't know how those keys are generated
you would need to run the javascript in a Node environment to write logging messages to a file, AFAIK
In short, I don't really know. 😂
The game's source code is so huge it would be hard to pinpoint what data would be relevant from a data analyzation standpoint.

@Lerbytech I don't think any of us are data engineers, so.... do you want to connect to the WebSocket? Or just record games? I don't really know which, but I recommend NodeJS and using a headless chrome like puppeteer to record the gameplay. Or if you just want to watch the WebSocket use a web-debugger like fiddler or the chrome devtools and that will let you inspect all the WebSocket traffic.
What do you hope to accomplish by doing this?

@Lerbytech Is this for an ML project? If so, I am really interested. I know how to inspect websocket data, but a github issue comment is really not the medium that i can use.

If you have some IM service or discord, I can message you there.

Reverse engineering the ws is relatively difficult, so you may want to create an extension that give you the readout of the data. I can help with that


repeated component with alias set property to late

When looping on a component with v-repeat, the property on the child component is set too late and:

it trigger a lot of warnings in the console if the property is required
it break every computed properties depending on it

This JsFiddle illustrate if (look into the debugger):
http://jsfiddle.net/noirbizarre/7u2e5yvb/1/

This has to do with the scoping issue of v-repeat. In 1.0.0 v-for will require explicitly passing in the props, and the type checks will work as expected.


Update cdt-gdb-vscode

At the moment this:
https://github.com/eclipse-cdt-cloud/cdt-cloud-blueprint/blob/6d0f6fbc1574d745062842ff736fcb64de842838/applications/electron/package.json#L135
is consuming cdt-gdb-vscode in a not very automated way in whole or in part because cdt-gdb-vscode is not published well.
Depends on: https://github.com/eclipse-cdt-cloud/cdt-gdb-vscode/issues/40

Closed via https://github.com/eclipse-cdt-cloud/cdt-cloud-blueprint/pull/33


GUAC-3337 Give hasExecuted a public interface

Not much to see here. This is so we can check if the data source controller has executed at least one complete fetch, including testing for the helper layer.

This looks ok, although I wonder if hasExecuted is being set properly - if the initial fetch fails, the hasExecuted flag won't get set, which won't prevent from adding ourselves as listeners for the notifications again should startListening be called again.  I suspect that a) the fetch doesn't often fail and b) if it did, it would be unusual for us to call startListening again, so this may not matter, but just a thought.

y'all are in a meeting rn, wyd

To be more truthy, we ought to have a separate flag for whether or not we've bound to listeners, and have that separate from hasExecuted, but it's so rarely a problem because fetching failures are rare.
We do a lot to make sure our data store is in a good state before we execute a fetch, and any other fetch failures are caused by developer failures, like bad fetch requests.


Support use with LTS 16.17.

Data.Primitive.UnliftedArray has been removed from primitive and
added to a new package, unlifted-primitive. But I'm pretty sure we're
not using either of these packages, so I just removed them both.
I guess Lift now already has an instance for Text. I'm not sure if
removing this loses backwards compatibility, but I don't think we need
to worry about that?

We've been using this in production excelsior for ages, so I guess I'm gonna merge it.


Where is this hosted?

Just a minor question. Where is this hosted? Thanks. :)

Currently the project is still a WIP.

okay thanks!
On Wed, Dec 21, 2022 at 9:17 PM Guillermo Flores V @.***>
wrote:

Currently the project is still a WIP.
—
Reply to this email directly, view it on GitHub
https://github.com/freeCodeCamp/classroom/issues/206#issuecomment-1362430845,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AT3BLYEZHOIBWARZGDY2FL3WOPP6BANCNFSM6AAAAAATCYXLIA
.
You are receiving this because you authored the thread.Message ID:
@.***>


gives me an error

Game name: vrchat
Engine version: i don't know
Platform: pc
Error message: Some data left
Stack trace:    in uTinyRipper.Lz4.Lz4DecodeStream.ReadBuffer(Stream stream, Int64 count)
in uTinyRipper.BundleFiles.BundleFileBlockReader.ReadEntry(Node entry)
in uTinyRipper.BundleFileScheme.ReadFileStreamData(Stream stream, Int64 basePosition, Int64 headerSize)
in uTinyRipper.BundleFileScheme.ReadScheme(Stream stream)
in uTinyRipper.GameCollection.ReadScheme(SmartStream stream, String filePath, String fileName)
in uTinyRipper.GameCollection.LoadScheme(String filePath, String fileName)
in uTinyRipper.GameStructure.ProcessPlatformStructure(GameStructureProcessor processor, PlatformGameStructure structure)
in uTinyRipper.GameStructure.Load(List1 pathes, LayoutInfo layinfo)    in uTinyRipper.GameStructure.Load(IEnumerable1 pathes, LayoutInfo layinfo)
in uTinyRipperGUI.MainWindow.LoadFiles(Object data)

Hello
uTinyRipper has been abandoned, but AssetRipper is actively being developed with the intent to recover Unity games as best as possible. I encourage you to help out if you're able and willing. Code contributions can be submitted via pull requests.
Development Progress
These are some of the notable things I've implemented since taking over development.

2020 - 6 support
APK, XAPK, and IPA reading with SharpZipLib
Dummy shaders (so everything isn't pink when you open the project)
IL2Cpp support via an integration of Cpp2IL
Better game platform support
Full script decompilation with the ILSpy decompiler
Cross-platform support
Multiple configuration options
Numerous performance improvements
Support for all asset types
Lightmap Recovery
High-quality 3D model export with SharpGltf
Prefab Outlining
Hybrid Script Export

This is not (at all) a comprehensive list. There are so many other things that could not be included here. They have all been made possible with the support of my generous sponsors.
Future Plans
This is a list of notable improvements planned for the future.

Mesh preview
Scene preview
Child State Machines
Dynamic Zip File Loading

Sponsorship
If you like the work I've done and want to see it continue, please consider sponsoring me. It really helps with development, more than you know.

GitHub Sponsors
Patreon
Paypal

A Patreon subscription gives access to AssetRipper Premium, which has additional features:

Static Mesh Separation
Asset Deduplication
User Defined Package Export


nanopolish call-methylation showing Segmentation fault (core dumped)

Dear jts,
I want to find methylation in nanopore raw data. I have a computer that has 64GB RAM, 18TB HDD and AMD® Ryzen 9 7950x 16-core processor × 32.
The nanopore data were analysed as mentioned in https://nanopolish.readthedocs.io/en/latest/quickstart_call_methylation.html.
But I got an error at last step "nanopolish call-methylation" as like following:
minimap2 -a -x map-ont /home/shaktikumar/RefGenome/hg38/hg38.fasta
merged_PAY16913_pass_b3c21dcf_78400604.fastq | samtools sort -T tmp -o merged_PAY16913_pass_b3c21dcf_78400604.sorted.bam
[M::mm_idx_gen::24.2681.54] collected minimizers
[M::mm_idx_gen::29.7951.81] sorted minimizers
[M::main::29.7951.81] loaded/built the index for 455 target sequence(s)
[M::mm_mapopt_update::30.7151.79] mid_occ = 728
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 455
[M::mm_idx_stat::31.1351.78] distinct minimizers: 100202295 (37.96% are singletons); average occurrences: 5.732; average spacing: 5.587; total length:<PHONE_NUMBER>
[M::worker_pipeline::155.4902.70] mapped 637633 sequences
[M::worker_pipeline::273.6902.84] mapped 632781 sequences
[M::worker_pipeline::379.0852.89] mapped 630885 sequences
[M::worker_pipeline::494.5922.92] mapped 636250 sequences
[M::worker_pipeline::599.0862.94] mapped 634315 sequences
[M::worker_pipeline::717.4812.96] mapped 629061 sequences
[M::worker_pipeline::820.6782.97] mapped 631298 sequences
[M::worker_pipeline::936.0922.97] mapped 634504 sequences
[M::worker_pipeline::1040.0432.98] mapped 633505 sequences
[M::worker_pipeline::1155.8742.98] mapped 635680 sequences
[M::worker_pipeline::1259.2612.99] mapped 635482 sequences
[M::worker_pipeline::1317.099*2.97] mapped 345380 sequences
[M::main] Version: 2.24-r1122
[M::main] CMD: minimap2 -a -x map-ont /home/shaktikumar/RefGenome/hg38/hg38.fasta merged_PAY16913_pass_b3c21dcf_78400604.fastq
[M::main] Real time: 1317.374 sec; CPU: 3915.111 sec; Peak RSS: 11.941 GB
[bam_sort_core] merging from 17 files and 1 in-memory blocks...
shaktikumar@shaktikumar-N7-B650E:/media/shaktikumar/seagate01/ngsdata/DrSwasti/SGPGI_Epitranscriptome/SGPGI/S_T_
27_08_2024_SGPGI/20240827_1825_1C_PAY16913_b3c21dcf$
shaktikumar@shaktikumar-N7-B650E:/media/shaktikumar/seagate01/ngsdata/DrSwasti/SGPGI_Epitranscriptome/SGPGI/S_T_27_08_2024_SGPGI/20240827_1825_1C_PAY16913_b3c21dcf$
shaktikumar@shaktikumar-N7-B650E:/media/shaktikumar/seagate01/ngsdata/DrSwasti/SGPGI_Epitranscriptome/SGPGI/S_T_27_08_2024_SGPGI/20240827_1825_1C_PAY16913_b3c21dcf$ samtools index merged_PAY16913_pass_b3c21dcf_78400604.sorted.bam
shaktikumar@shaktikumar-N7-B650E:/media/shaktikumar/seagate01/ngsdata/DrSwasti/SGPGI_Epitranscriptome/SGPGI/S_T_27_08_2024_SGPGI/20240827_1825_1C_PAY16913_b3c21dcf$ nanopolish call-methylation -t 8 -r merged_PAY16913_pass_b3c21dcf_78400604.fastq -b merged_PAY16913_pass_b3c21dcf_78400604.sorted.bam -g /home/shaktikumar/RefGenome/hg38/hg38.fasta > methylation_calls.tsv
Segmentation fault (core dumped)

Hi @shakti83kumar,
It looks like you are trying to call rna modifications using recent (R10.4.1?) data, which nanopolish does not support. If this is the case, I suggest using dorado instead.
Jared

How did you know Sir?

In the screenshots you provided the directory said "Epitranscriptomics"

Ok ...
Thanks jts Sir.